Tim Hardy

I was asked to help delete 16,980 messages from an IMAP spam folder the other day. No email client could handle it without crashing. Even mutt choked after several hours of valiantly struggling.

Python to the rescue. Rather than write a script to do this I ran each command from the python shell. It’s a very addictive way of working because you get instant feedback.

import getpass, imaplib
M = imaplib.IMAP4_SSL("imap.gmail.com")
M.login("yourusername@gmail.com", getpass.getpass())


Now we’re in. Let’s see what directories exist.

M.list()


Pick the offending directory.

M.select("[Gmail]/Spam")


View the messages.

typ, data = M.search(None, 'ALL')
for num in data[0].split():
....typ, data = M.fetch(num, '(RFC822)')
....print 'Message %s\n%s\n' % (num, data[0][1])


Now delete them all and close the connection to the mailserver. To delete a message in IMAP, you need to set the delete flag on it then expunge the folder.

typ, data = M.search(None, 'ALL')
for num in data[0].split():
....M.store(num, '+FLAGS', '\\Deleted')
M.expunge()
M.close()
M.logout()


(To be fair to google I must point out that gmail was not the offending mailserver although I’ve used them in the sample code above.)

I’ve recently built a new desktop machine - the subject of another post when I get time - and because the hardware is fairly recent I decided to try Ubuntu which I knew would support it rather than switch to an unstable branch of Debian.

My first impressions of Hardy Heron - Ubuntu’s latest release; I’m using the variant with the KDE desktop - are very positive. The installer is excellent and the hardware detection very impressive. As a comparison, it took me the better part of a day to install a fairly vanilla Windows XP Pro SP2 on the first partition of the box. Nearly every driver install required a reboot as did many of the security updates. By contrast, it took less than an hour to download and install Kubuntu and the installation includes a far greater range of packages than I have on the windows partition.

The one fly in the ointment has been a problem with K3b. If I started with a blank CD in the drive, all was fine but launching it with either an empty drive or a blank DVD caused the process to hang for a long while and finally launch with the warning:

“No CD/DVD writer found.
K3b did not find an optical writing device in your system. Thus, you will not be able to burn CDs or DVDs. However, you can still use other K3b features like audio track extraction or audio transcoding or ISO9660 image creation.”

After this the DVD drive was disabled, making it impossible to even eject until the machine had been shutdown completely and restarted.

Both growisofs and wodim worked from the command line so the problem had to be in the KDE application. Launching K3B from the console reveals debug information and showed the hang up was occurring after the step checking for RAW_R96R:

...
(K3bDevice::Device) /dev/scd0: checking for RAW_R96R

After a long pause the process would continue:

(K3bDevice::ScsiCommand) failed:
                           command:    GET PERFORMANCE (ac)
                           errorcode:  72
                           sense key:  NO SENSE (2)
                           asc:        0
                           ascq:       3
(K3bDevice::Device) /dev/scd0: GET PERFORMANCE length det failed.
(K3bDevice::ScsiCommand) failed:
                           command:    MODE SENSE (5a)
                           errorcode:  0
                           sense key:  NO SENSE (2)
                           asc:        0
                           ascq:       0
(K3bDevice::Device) /dev/scd0: MODE SENSE length det failed.
(K3bDevice::ScsiCommand) failed:
                           command:    MODE SENSE (5a)
                           errorcode:  0
                           sense key:  NO SENSE (2)
                           asc:        0
                           ascq:       0
(K3bDevice::Device) /dev/scd0: MODE SENSE with real length 65535 failed.
(K3bDevice::Device) could not open device /dev/scd0 for reading
                    (No such device or address)
Error while retrieving capabilities.
Could not initialize device /dev/scd0
Could not resolve /dev/scd0
/dev/scd0 resolved to /dev/scd0
(K3bDevice::Device) could not open device /dev/scd0 for reading
                    (No such file or directory)
could not open device /dev/scd0 (No such file or directory)
Devices:
------------------------------
kdecore (KAction): WARNING: KActionCollection::operator+=(): function is severely deprecated.
removing udi /org/freedesktop/Hal/devices/pci_8086_2922_scsi_host_0_scsi_device_                                                                                                                                                 lun0_scsi_generic
removing udi /org/freedesktop/Hal/devices/pci_8086_2922_scsi_host_0_scsi_device_                                                                                                                                                 lun0
removing udi /org/freedesktop/Hal/devices/pci_8086_2922_scsi_host_0
removing udi /org/freedesktop/Hal/devices/storage_model_DVDRW_LH_20A1S
Unmapping udi /org/freedesktop/Hal/devices/storage_model_DVDRW_LH_20A1S from dev                                                                                                                                                 ice /dev/scd0

dmesg reveals the following occuring

May  1 23:52:15 mingus kernel: [ 3580.143800]          cdb ac 00 00 00 00 00 00 00  00 01 03 00 00 00 00 00
May  1 23:52:15 mingus kernel: [ 3580.143801]          res 40/00:03:00:00:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
May  1 23:52:15 mingus kernel: [ 3580.455532] ata6: soft resetting link
May  1 23:52:15 mingus kernel: [ 3580.619411] ata6: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
May  1 23:52:45 mingus kernel: [ 3610.751976] ata6.00: qc timeout (cmd 0xa1)
May  1 23:52:45 mingus kernel: [ 3610.751985] ata6.00: failed to IDENTIFY (I/O error, err_mask=0x5)
May  1 23:52:45 mingus kernel: [ 3610.751993] ata6: failed to recover some devices, retrying in 5 secs
May  1 23:52:50 mingus kernel: [ 3615.752086] ata6: hard resetting link
May  1 23:52:56 mingus kernel: [ 3621.263794] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:53:00 mingus kernel: [ 3625.752312] ata6: hard resetting link
May  1 23:53:06 mingus kernel: [ 3631.268013] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:53:11 mingus kernel: [ 3635.804490] ata6: hard resetting link
May  1 23:53:16 mingus kernel: [ 3641.316197] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:53:46 mingus kernel: [ 3670.813264] ata6: limiting SATA link speed to 1.5 Gbps
May  1 23:53:46 mingus kernel: [ 3670.813266] ata6: hard resetting link
May  1 23:53:51 mingus kernel: [ 3675.821377] ata6.00: disabled
May  1 23:53:51 mingus kernel: [ 3675.821404] ata6: hard resetting link
May  1 23:53:57 mingus kernel: [ 3681.736761] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:54:01 mingus kernel: [ 3685.825584] ata6: hard resetting link
May  1 23:52:15 mingus kernel: [ 3580.143800]          cdb ac 00 00 00 00 00 00 00  00 01 03 00 00 00 00 00
May  1 23:52:15 mingus kernel: [ 3580.143801]          res 40/00:03:00:00:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
May  1 23:52:15 mingus kernel: [ 3580.455532] ata6: soft resetting link
May  1 23:52:15 mingus kernel: [ 3580.619411] ata6: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
May  1 23:52:45 mingus kernel: [ 3610.751976] ata6.00: qc timeout (cmd 0xa1)
May  1 23:52:45 mingus kernel: [ 3610.751985] ata6.00: failed to IDENTIFY (I/O error, err_mask=0x5)
May  1 23:52:45 mingus kernel: [ 3610.751993] ata6: failed to recover some devices, retrying in 5 secs
May  1 23:52:50 mingus kernel: [ 3615.752086] ata6: hard resetting link
May  1 23:52:56 mingus kernel: [ 3621.263794] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:53:00 mingus kernel: [ 3625.752312] ata6: hard resetting link
May  1 23:53:06 mingus kernel: [ 3631.268013] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:53:11 mingus kernel: [ 3635.804490] ata6: hard resetting link
May  1 23:53:16 mingus kernel: [ 3641.316197] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:53:46 mingus kernel: [ 3670.813264] ata6: limiting SATA link speed to 1.5 Gbps
May  1 23:53:46 mingus kernel: [ 3670.813266] ata6: hard resetting link
May  1 23:53:51 mingus kernel: [ 3675.821377] ata6.00: disabled
May  1 23:53:51 mingus kernel: [ 3675.821404] ata6: hard resetting link
May  1 23:53:57 mingus kernel: [ 3681.736761] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:54:01 mingus kernel: [ 3685.825584] ata6: hard resetting link
May  1 23:54:07 mingus kernel: [ 3691.752971] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:54:11 mingus kernel: [ 3695.841797] ata6: hard resetting link
May  1 23:54:17 mingus kernel: [ 3701.757192] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:54:46 mingus kernel: [ 3730.858562] ata6: limiting SATA link speed to 1.5 Gbps
May  1 23:54:46 mingus kernel: [ 3730.858565] ata6: hard resetting link
May  1 23:54:51 mingus kernel: [ 3735.884188] ata6: EH complete
May  1 23:54:51 mingus kernel: [ 3735.884287] ata6.00: detaching (SCSI 5:0:0:0)

It looks like the bug is probably in the libsata code failing to handle an error rather than K3B but I haven’t had a chance to investigate further.

As a quick workaround, download the k3b source and hack bool K3bDevice::Device::getSupportedWriteSpeedsViaGP to return false immediately and shortcircuit the call to ioctl that locks the device. The code will fall back to setting a maximum speed determined by the previous call to getSupportedWriteSpeedsVia2A.

I’ve recently discovered an excellent new podcast on software engineering - Software Engineering Radio. I’ve long been a fan of IT Conversations and In Our Time and this is a very welcome addition to my listening schedule.

As I client I’ve been using my Nokia N95 after upgrading the firmware to v20.0.015. The upgraded phone has no difficulty addressing memory cards larger than 4G, vastly improved battery life while playing mp3s, no longer crashes from running out of memory (thanks to on demand paging which also cuts the boot time significantly) and has a podcast client built in that remembers where I left off a particular episode. This is ideal if, like me, you use podcasts to claw back otherwise wasted time and so often digest episodes in several minute chunks.

Oh, and you can sync up your subscriptions and download new episodes direct to the player using wireless which makes it extremely convenient.

I’m working through the Software Engineering Radio backlist at the moment and have particularly enjoyed the following episodes on game development, Erlang, Lisp and an interview with Erich Gamma. You know something must be good when you find yourself looking forward to doing the washing up because it will give you another ten minutes to listen to more of the latest episode. For anyone interested in software development, I’d heartily recommend it.

In file included from vhook/imlib2.c:48:
/home/tim/src/ffmpeg/libavformat/framehook.h:25:2: warning: #warning VHOOK is deprecated. Please help porting libmpcodecs or a better filter system to FFmpeg instead of wasting your time writing new filters for this crappy one.

Starting with Bjarne Stroustrup’s observation “our civilization runs on software”, Grady Booch offered a thought provoking overview of the history and promise of software at this year’s BCS Turing lecture, taking listeners from the austere beauty of Alan Turing’s 1930s thought experiments through to “the rise of the machines” in 2030.

Booch is an interesting, relaxed and witty speaker, whose asides on the superiority of OS X to Windows, George Bush and Google (”Am I the only one who thinks there’s a company in desperate need of some adult supervision?”) provided comic relief in an at times informationally dense speech.

One point that intrigued me was his observation that much of the history of computing is unrecorded, existing only in the “tribal memory” of the greybeards. He foresees the emergence of both software artists and historians who might translate and record some of the strange beauty of code for non-programmers as well as formally archiving a form of communication in danger of vanishing with the death of its authors.

The full lecture is available as a recording from the link below and is well worth watching.

A few extracts from a great two-part interview with Bjarne Stroustrup: Technology Review: The Problem with Programming,
More Trouble with Programming.

On over-abstraction:

Software developers have neutralized the astounding performance of modern computer hardware by adding layer upon layer of overelaborate [software] abstractions. We seem to have hit the limits of linear speedup for hardware, but in many cases, we could win a couple of orders of magnitude back from the software.

On “the next big thing”:

I don’t know what the next major conceptual shift will be, but I bet that it will somehow be related to the management of concurrency. As programmers, we have been notoriously bad at thinking about lots of things happening simultaneously, and soon our everyday computers will have 32 cores.

On .NET and the JRE:

I want my software to run everywhere it makes sense to run it. I also want to be able to change suppliers of parts of my system if the suppliers are not the best. Obviously, suppliers of huge integrated systems, such as .Net and Java, see things differently. Their claim is that what they provide is worth more to users than independence. Sometimes they are right, and of course some degree of integration is necessary: you cannot write a complete application of any realistic size without introducing some system dependencies. The question is how deeply integrated into the application those system dependencies are. I prefer the application to be designed conceptually in isolation from the underlying system, with an explicitly defined interface to “the outer world,” and then integrated through a thin layer of interface code.

Well worth the read.

My father told me an anecdote about a patient who, while clearly very ill, showed a baffling lack of symptoms. To the first doctor he claimed everything was perfectly normal. To the second, he claimed that he’d noticed no change to his normal daily routine. It wasn’t until a the third doctor to see him asked him to step through a “normal” day that a missing key symptom became apparent: “The alarm goes off at 7.30, I get out of bed, go to the bathroom, throw up…”

This isn’t meant as an argument for using debuggers when coding but rather a warning always to challenge assumptions and to fix problems when they occur. Routine blinds us to everything. This can be a blessing in day-to-day living but it’s a curse when problems cascade and the root cause is obscured by habit.

If there’s an error in your java code, Eclipse sticks a marker in the gutter with a light-bulb icon. Clicking on the marker or pressing Ctl+1 with the cursor in the offending line brings up a helpful pop-up of suggested fixes for the problem.

I’ve always assumed that this was called Quick Fix. If you right click on the marker, the option is labelled Quick Fix. However, after struggling for a long while to understand why some Quick Fix code I wrote wasn’t showing up in this menu, I’ve come to the conclusion that this is actually Quick Assist and that the label is wrong (I’ve flagged this as a bug).

To launch Quick Fix you need instead to right click on a Warning/Error in the Problems View. Quick Fix and Quick Assist are in fact two different things with confusingly similar names and functionality.

Internally appropriate Quick Fixes are calculated by a class implementing IMarkerResolutionGenerator that extends org.eclipse.ui.ide.markerResolution. The fixes themselves are contained in classes implementing either IMarkerResolution or IMarkerResolution2. By contrast, Quick Assists are calculated by a class extending org.eclipse.jdt.ui.quickAssistProcessors and implement IQuickAssistProcessor. The classes containing the assists implement IJavaCompletionProposal.

So, if you’re developing a plug-in and cannot work out why your Quick Fixes are not showing up, take a closer look. Maybe like me you’re checking in the wrong place and what you meant to write was actually a Quick Assist.

I’ve been coding exclusively in Java for the last few weeks. However, when I needed to write a quick program to crunch some numbers the other day, I decided to write it in C++ for the change. Since I’ve also been working exclusively within Eclipse, I decided to use a simple text editor for this and gcc from the command line.

The experience made me reflect a little on the importance of good tools.

In Contributing to Eclipse, Kent Beck and Erich Gamma speak of the compiler moving from being a separate tool to “air - something vital that’s always around”. While using the JDT, the compiler is constantly running in the background, warning you of potential errors as you type each line of code. The IDE also makes suggestions for fixes that save much typing. In this collaborative model, the compiler/editor is like a helpful colleague looking over your shoulder and making gentle suggestions from time to time. By contrast, the model of finishing the code, trying to compile it and working through the often opaque error warnings thrown by a c++ compiler is more like facing a difficult interview. The rhythm of work is completely different.

Doug Schaefer, project lead for the Eclipse CDT, explains in this Callisto podcast (direct link to mp3) that the simplicity of Java makes the JDT experience possible but that it would be prohibitively expensive to do the same with CDT.

Another thing I missed was JUnit. At the moment, I frequently work outside my comfort zone, learning through discovery and error and unit testing gives me the confidence to do this and to refactor. The task at hand was so brief it wasn’t worth investing the time to explore the xUnit variants for C++ but I intend to do so the moment I need to write more substantial code in that language.

Eclipse 3.2 is out but I haven’t had much of a chance to look at it yet because I’ve spent the afternoon trying to get it to install.

The GTK focus bug from 3.1 is still present (the problem lies with Debian’s use of version numbers) so you need to download the source, patch it and compile it for your own platform. (Instructions here.)

Compilation proved a little trickier than it was for 3.1. As before you need to edit plugins/org.eclipse.swt/Eclipse SWT/gtk/org/eclipse/swt/widgets/Shell.java - feel free to use my patch from the root of your unzipped source:

$ patch -p0 < debGTKBugPatch

I couldn’t get the default build to work but finally got things compiling (ignoring literally hundreds of warnings about non-type safe generic code) by editing the build.xml file and changing javacSource and javacTarget to 1.5 (I only have Java 1.5 on my machine: it should be possible to compile at least some of the source using Java 1.4 but I haven’t tried it)

<property name="javacSource" value="1.5" />
<property name="javacTarget" value="1.5" />

Finally start the build process and go and do something else for a while (on my ageing P4 1.8 it takes close to 50 minutes to build).

$ export ANT_HOME=/usr/bin/ant
$ export JAVA_HOME=/usr/
$ chmod +x build
$ ./build -os linux -ws gtk -arch x86 -java5home /usr/bin/


The custom built version will be in result/linux-gtk-x86-sdk.tar.gz. Extract it into another directory and run eclipse from there.

These notes explain how I got it to work on my machine. Your setup may differ.