Retrieving Rapidshare Files with Python

January 2nd, 2009

A cursory google search will reveal several scripts for retrieving rapidshare files using python, but each one I’ve seen delegates the actual retrieval to wget.

This is not necessary.

Rapidshare uses basic authentication to identify logged in members and urllib2 can handle this easily.

The following method would do the trick without the need to call external executables:

def rapidget(url. login, password):
    "Retrieve files from rapidshare using only python"
    request = urllib2.Request(url)
    base64string = base64.encodestring('%s:%s' % (login, password))[:-1]
    request.add_header("Authorization", "Basic %s" % base64string)
    i = url.rfind('/')
    filename = url[i+1:]
    print url, "->", filename
    file = open(filename, 'wb')
    handle = urllib2.urlopen(request)
    buffer = ''
    buffersize = 1024*1024
    while True:
        buffer = handle.read(buffersize)
        if not buffer:
            handle.close()
            file.close()
            break
        file.write(buffer)
        buffer = ''
        print '.',

This assumes, of course, that you have an account at rapidshare.

Posted in Internet, Programming, Uncategorized, howto, python | Comments Off

Is the iPlayer a Trojan Horse?

December 31st, 2008

I won’t be joining the celebrations around the launch of the BBC’s iPlayer on Mac and Linux.

The encroachment into the network of broadcasting corporations such as the BBC should be vigorously resisted both as a tremendous waste of bandwidth by a company that already enjoys a monopoly on huge swathes of the spectrum and as a step towards the licensing of internet access.

UK readers sensible enough not to own a television will have first hand experience of the Gestapo-like tactics of the BBC licensing authorities whose regular, nasty, intimidatory letters misleadingly and illegally threaten prosecution to anyone found using equipment capable of receiving a television signal including “computers connected to the internet.” The more organisations like the BBC pollute the web with their output, the stronger the calls to extend the license to cover access to the internet.

Already the iPlayer is being tested as a justification for bringing a tiered internet into place.

Combine that with a quixotic and sinister plan to introduce cinema style ratings to websites being considered and we have all the makings of Chinese-style censorship.

Paranoid? Perhaps. But I do live under a government planning on tracking everyone’s calls, emails, texts and internet use.

Posted in Culture, General IT, Internet, Linux, News, Politics | 1 Comment »

Governments are a Conspiracy of the Rich

December 27th, 2008

What justice is there in this: that a nobleman, a goldsmith, a banker, or any other man, that either does nothing at all, or, at best, is employed in things that are of no use to the public, should live in great luxury and splendour upon what is so ill acquired, and a mean man, a carter, a smith, or a ploughman, that works harder even than the beasts themselves, and is employed in labours so necessary, that no commonwealth could hold out a year without them, can only earn so poor a livelihood and must lead so miserable a life, that the condition of the beasts is much better than theirs? For as the beasts do not work so constantly, so they feed almost as well, and with more pleasure, and have no anxiety about what is to come, whilst these men are depressed by a barren and fruitless employment, and tormented with the apprehensions of want in their old age; since that which they get by their daily labour does but maintain them at present, and is consumed as fast as it comes in, there is no overplus left to lay up for old age.

“Is not that government both unjust and ungrateful, that is so prodigal of its favours to those that are called gentlemen, or goldsmiths, or such others who are idle, or live either by flattery or by contriving the arts of vain pleasure, and, on the other hand, takes no care of those of a meaner sort, such as ploughmen, colliers, and smiths, without whom it could not subsist? But after the public has reaped all the advantage of their service, and they come to be oppressed with age, sickness, and want, all their labours and the good they have done is forgotten, and all the recompense given them is that they are left to die in great misery. The richer sort are often endeavouring to bring the hire of labourers lower, not only by their fraudulent practices, but by the laws which they procure to be made to that effect, so that though it is a thing most unjust in itself to give such small rewards to those who deserve so well of the public, yet they have given those hardships the name and colour of justice, by procuring laws to be made for regulating them.

“Therefore I must say that, as I hope for mercy, I can have no other notion of all the other governments that I see or know, than that they are a conspiracy of the rich, who, on pretence of managing the public, only pursue their private ends, and devise all the ways and arts they can find out; first, that they may, without danger, preserve all that they have so ill-acquired, and then, that they may engage the poor to toil and labour for them at as low rates as possible, and oppress them as much as they please; and if they can but prevail to get these contrivances established by the show of public authority, which is considered as the representative of the whole people, then they are accounted laws

Sir Thomas More, Utopia, 1516

Posted in Culture, Politics | Comments Off

Deleting Messages from an IMAP Folder Using Python

August 16th, 2008

I was asked to help delete 16,980 messages from an IMAP spam folder the other day. No email client could handle it without crashing. Even mutt choked after several hours of valiantly struggling.

Python to the rescue. Rather than write a script to do this I ran each command from the python shell. It’s a very addictive way of working because you get instant feedback.

import getpass, imaplib
M = imaplib.IMAP4_SSL("imap.gmail.com")
M.login("yourusername@gmail.com", getpass.getpass())

Now we’re in. Let’s see what directories exist.

M.list()

Pick the offending directory.

M.select("[Gmail]/Spam")

View the messages.

typ, data = M.search(None, 'ALL')
for num in data[0].split():
....typ, data = M.fetch(num, '(RFC822)')
....print 'Message %s\n%s\n' % (num, data[0][1])

Now delete them all and close the connection to the mailserver. To delete a message in IMAP, you need to set the delete flag on it then expunge the folder.

typ, data = M.search(None, 'ALL')
for num in data[0].split():
....M.store(num, '+FLAGS', '\\Deleted')
M.expunge()
M.close()
M.logout()

(To be fair to google I must point out that gmail was not the offending mailserver although I’ve used them in the sample code above.)

Posted in Internet, Programming, howto, python | 5 Comments »

Marx the Poet

August 3rd, 2008

It is absolutely clear that, by his activity, man changes the forms of the materials of nature in such a way as to make them useful to him. The form of wood, for instance, is altered if a table is made out of it. Nevertheless the table continues to be wood, an ordinary, sensuous thing. But as soon as it emerges as a commodity, it changes into a thing which transcends sensuousness. It not only stands with its feet on the ground, but, in relation to all other commodities, it stands on its head, and evolves out of its wooden brain grotesque ideas, far more wonderful than if it were to begin dancing of its own free will.

Das Kapital, Karl Marx

Posted in Culture, Writing | Comments Off

Patching K3B on Kubuntu Hardy Heron

May 6th, 2008

I’ve recently built a new desktop machine – the subject of another post when I get time – and because the hardware is fairly recent I decided to try Ubuntu which I knew would support it rather than switch to an unstable branch of Debian.

My first impressions of Hardy Heron – Ubuntu’s latest release; I’m using the variant with the KDE desktop – are very positive. The installer is excellent and the hardware detection very impressive. As a comparison, it took me the better part of a day to install a fairly vanilla Windows XP Pro SP2 on the first partition of the box. Nearly every driver install required a reboot as did many of the security updates. By contrast, it took less than an hour to download and install Kubuntu and the installation includes a far greater range of packages than I have on the windows partition.

The one fly in the ointment has been a problem with K3b. If I started with a blank CD in the drive, all was fine but launching it with either an empty drive or a blank DVD caused the process to hang for a long while and finally launch with the warning:

“No CD/DVD writer found.
K3b did not find an optical writing device in your system. Thus, you will not be able to burn CDs or DVDs. However, you can still use other K3b features like audio track extraction or audio transcoding or ISO9660 image creation.”

After this the DVD drive was disabled, making it impossible to even eject until the machine had been shutdown completely and restarted.

Both growisofs and wodim worked from the command line so the problem had to be in the KDE application. Launching K3B from the console reveals debug information and showed the hang up was occurring after the step checking for RAW_R96R:

...
(K3bDevice::Device) /dev/scd0: checking for RAW_R96R

After a long pause the process would continue:

(K3bDevice::ScsiCommand) failed:
                           command:    GET PERFORMANCE (ac)
                           errorcode:  72
                           sense key:  NO SENSE (2)
                           asc:        0
                           ascq:       3
(K3bDevice::Device) /dev/scd0: GET PERFORMANCE length det failed.
(K3bDevice::ScsiCommand) failed:
                           command:    MODE SENSE (5a)
                           errorcode:  0
                           sense key:  NO SENSE (2)
                           asc:        0
                           ascq:       0
(K3bDevice::Device) /dev/scd0: MODE SENSE length det failed.
(K3bDevice::ScsiCommand) failed:
                           command:    MODE SENSE (5a)
                           errorcode:  0
                           sense key:  NO SENSE (2)
                           asc:        0
                           ascq:       0
(K3bDevice::Device) /dev/scd0: MODE SENSE with real length 65535 failed.
(K3bDevice::Device) could not open device /dev/scd0 for reading
                    (No such device or address)
Error while retrieving capabilities.
Could not initialize device /dev/scd0
Could not resolve /dev/scd0
/dev/scd0 resolved to /dev/scd0
(K3bDevice::Device) could not open device /dev/scd0 for reading
                    (No such file or directory)
could not open device /dev/scd0 (No such file or directory)
Devices:
------------------------------
kdecore (KAction): WARNING: KActionCollection::operator+=(): function is severely deprecated.
removing udi /org/freedesktop/Hal/devices/pci_8086_2922_scsi_host_0_scsi_device_                                                                                                                                                 lun0_scsi_generic
removing udi /org/freedesktop/Hal/devices/pci_8086_2922_scsi_host_0_scsi_device_                                                                                                                                                 lun0
removing udi /org/freedesktop/Hal/devices/pci_8086_2922_scsi_host_0
removing udi /org/freedesktop/Hal/devices/storage_model_DVDRW_LH_20A1S
Unmapping udi /org/freedesktop/Hal/devices/storage_model_DVDRW_LH_20A1S from dev                                                                                                                                                 ice /dev/scd0

dmesg reveals the following occuring

May  1 23:52:15 mingus kernel: [ 3580.143800]          cdb ac 00 00 00 00 00 00 00  00 01 03 00 00 00 00 00
May  1 23:52:15 mingus kernel: [ 3580.143801]          res 40/00:03:00:00:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
May  1 23:52:15 mingus kernel: [ 3580.455532] ata6: soft resetting link
May  1 23:52:15 mingus kernel: [ 3580.619411] ata6: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
May  1 23:52:45 mingus kernel: [ 3610.751976] ata6.00: qc timeout (cmd 0xa1)
May  1 23:52:45 mingus kernel: [ 3610.751985] ata6.00: failed to IDENTIFY (I/O error, err_mask=0x5)
May  1 23:52:45 mingus kernel: [ 3610.751993] ata6: failed to recover some devices, retrying in 5 secs
May  1 23:52:50 mingus kernel: [ 3615.752086] ata6: hard resetting link
May  1 23:52:56 mingus kernel: [ 3621.263794] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:53:00 mingus kernel: [ 3625.752312] ata6: hard resetting link
May  1 23:53:06 mingus kernel: [ 3631.268013] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:53:11 mingus kernel: [ 3635.804490] ata6: hard resetting link
May  1 23:53:16 mingus kernel: [ 3641.316197] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:53:46 mingus kernel: [ 3670.813264] ata6: limiting SATA link speed to 1.5 Gbps
May  1 23:53:46 mingus kernel: [ 3670.813266] ata6: hard resetting link
May  1 23:53:51 mingus kernel: [ 3675.821377] ata6.00: disabled
May  1 23:53:51 mingus kernel: [ 3675.821404] ata6: hard resetting link
May  1 23:53:57 mingus kernel: [ 3681.736761] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:54:01 mingus kernel: [ 3685.825584] ata6: hard resetting link
May  1 23:52:15 mingus kernel: [ 3580.143800]          cdb ac 00 00 00 00 00 00 00  00 01 03 00 00 00 00 00
May  1 23:52:15 mingus kernel: [ 3580.143801]          res 40/00:03:00:00:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
May  1 23:52:15 mingus kernel: [ 3580.455532] ata6: soft resetting link
May  1 23:52:15 mingus kernel: [ 3580.619411] ata6: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
May  1 23:52:45 mingus kernel: [ 3610.751976] ata6.00: qc timeout (cmd 0xa1)
May  1 23:52:45 mingus kernel: [ 3610.751985] ata6.00: failed to IDENTIFY (I/O error, err_mask=0x5)
May  1 23:52:45 mingus kernel: [ 3610.751993] ata6: failed to recover some devices, retrying in 5 secs
May  1 23:52:50 mingus kernel: [ 3615.752086] ata6: hard resetting link
May  1 23:52:56 mingus kernel: [ 3621.263794] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:53:00 mingus kernel: [ 3625.752312] ata6: hard resetting link
May  1 23:53:06 mingus kernel: [ 3631.268013] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:53:11 mingus kernel: [ 3635.804490] ata6: hard resetting link
May  1 23:53:16 mingus kernel: [ 3641.316197] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:53:46 mingus kernel: [ 3670.813264] ata6: limiting SATA link speed to 1.5 Gbps
May  1 23:53:46 mingus kernel: [ 3670.813266] ata6: hard resetting link
May  1 23:53:51 mingus kernel: [ 3675.821377] ata6.00: disabled
May  1 23:53:51 mingus kernel: [ 3675.821404] ata6: hard resetting link
May  1 23:53:57 mingus kernel: [ 3681.736761] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:54:01 mingus kernel: [ 3685.825584] ata6: hard resetting link
May  1 23:54:07 mingus kernel: [ 3691.752971] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:54:11 mingus kernel: [ 3695.841797] ata6: hard resetting link
May  1 23:54:17 mingus kernel: [ 3701.757192] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:54:46 mingus kernel: [ 3730.858562] ata6: limiting SATA link speed to 1.5 Gbps
May  1 23:54:46 mingus kernel: [ 3730.858565] ata6: hard resetting link
May  1 23:54:51 mingus kernel: [ 3735.884188] ata6: EH complete
May  1 23:54:51 mingus kernel: [ 3735.884287] ata6.00: detaching (SCSI 5:0:0:0)

It looks like the bug is probably in the libsata code failing to handle an error rather than K3B but I haven’t had a chance to investigate further.

As a quick workaround, download the k3b source and hack bool K3bDevice::Device::getSupportedWriteSpeedsViaGP to return false immediately and shortcircuit the call to ioctl that locks the device. The code will fall back to setting a maximum speed determined by the previous call to getSupportedWriteSpeedsVia2A.

Posted in Linux, Programming, howto | 3 Comments »

Podcasts for Software Engineers (and the Perfect Player: the Nokia N95)

April 27th, 2008

I’ve recently discovered an excellent new podcast on software engineering – Software Engineering Radio. I’ve long been a fan of IT Conversations and In Our Time and this is a very welcome addition to my listening schedule.

As I client I’ve been using my Nokia N95 after upgrading the firmware to v20.0.015. The upgraded phone has no difficulty addressing memory cards larger than 4G, vastly improved battery life while playing mp3s, no longer crashes from running out of memory (thanks to on demand paging which also cuts the boot time significantly) and has a podcast client built in that remembers where I left off a particular episode. This is ideal if, like me, you use podcasts to claw back otherwise wasted time and so often digest episodes in several minute chunks.

Oh, and you can sync up your subscriptions and download new episodes direct to the player using wireless which makes it extremely convenient.

I’m working through the Software Engineering Radio backlist at the moment and have particularly enjoyed the following episodes on game development, Erlang, Lisp and an interview with Erich Gamma. You know something must be good when you find yourself looking forward to doing the washing up because it will give you another ten minutes to listen to more of the latest episode. For anyone interested in software development, I’d heartily recommend it.

Posted in General IT, Nokia, Programming, Symbian | 2 Comments »

Extraordinary Customer Service from Amazon

March 8th, 2008

A DVD player I bought from amazon developed a fault.

I called them on Monday at lunchtime to report the problem.

Tuesday morning, they delivered a replacement before 8am. Tuesday evening a second courier came round to collect the faulty one and return it to amazon.

I’ve long been impressed by amazon but this time they have surpassed themselves.

Posted in Announcements, Culture, Personal | Comments Off

56K Nostalgia

November 27th, 2007

What the lights on a modem mean:

TR Terminal Ready Illuminated when terminal software is activated
HS High Speed Illuminated during K56flex connections (32.000 to 56.000 bps)
CD Carrier Detect The modem is connected with another modem.
AA Auto Answer The modem is ready to answer incoming calls automatically.
OH Off Hook The modem has picked up the phone line, just like taking your phone handset “off hook.”
SD Send Data Flashes as data are sent.
RD Receive Data Flashes as data are received.
PWR Power The modem is receiving power.

Posted in Uncategorized | Comments Off

Using Mutt with Exim4 to Send Mail via Smarthost over SSL (TLS) Connection on Port 465

November 26th, 2007

One day I’ll learn pine and I’m sure I’ll love it but for now I’m hooked on mutt.

Getting mutt to use exim4 for delivery is as simple as adding set sendmail="/usr/sbin/exim4" to your ~/.muttrc. Getting exim4 working as a smarthost to authenticate over port 465 is far harder.

The solution is to use a tunnel as outlined in this excellent post: exim4 with ssmtp on Debian.

Before finding this answer, I also discovered a very clear post on testing an smtp auth connection here and a useful tutorial on setting up and testing exim, both of which may be helpful to other people.

Posted in Internet, Linux, Uncategorized, howto | Comments Off

« Older Entries
Newer Entries »