Tim Hardy

I was asked to help delete 16,980 messages from an IMAP spam folder the other day. No email client could handle it without crashing. Even mutt choked after several hours of valiantly struggling.

Python to the rescue. Rather than write a script to do this I ran each command from the python shell. It’s a very addictive way of working because you get instant feedback.

import getpass, imaplib
M = imaplib.IMAP4_SSL("imap.gmail.com")
M.login("yourusername@gmail.com", getpass.getpass())


Now we’re in. Let’s see what directories exist.

M.list()


Pick the offending directory.

M.select("[Gmail]/Spam")


View the messages.

typ, data = M.search(None, 'ALL')
for num in data[0].split():
....typ, data = M.fetch(num, '(RFC822)')
....print 'Message %s\n%s\n' % (num, data[0][1])


Now delete them all and close the connection to the mailserver. To delete a message in IMAP, you need to set the delete flag on it then expunge the folder.

typ, data = M.search(None, 'ALL')
for num in data[0].split():
....M.store(num, '+FLAGS', '\\Deleted')
M.expunge()
M.close()
M.logout()


(To be fair to google I must point out that gmail was not the offending mailserver although I’ve used them in the sample code above.)

One day I’ll learn pine and I’m sure I’ll love it but for now I’m hooked on mutt.

Getting mutt to use exim4 for delivery is as simple as adding set sendmail="/usr/sbin/exim4" to your ~/.muttrc. Getting exim4 working as a smarthost to authenticate over port 465 is far harder.

The solution is to use a tunnel as outlined in this excellent post: exim4 with ssmtp on Debian.

Before finding this answer, I also discovered a very clear post on testing an smtp auth connection here and a useful tutorial on setting up and testing exim, both of which may be helpful to other people.

I have finally tired of mailsnare’s poor customer service. For a long while, the only way to get a response to a ticket was to post on the forums and publicly shame them into responding. Then they deleted the forums.

After trying several candidates, I have finally settled on webmail.us.

There are IMAP providers out there offering more bells and whistles but webmail.us advertises 99.99% uptime and offers 24 hour support by email, web or phone. With that kind of service, I can live without cutting edge, alpha-geek features that aren’t much use when the email server has been down for three days and no one is responding to requests for service…

Migrating several years worth of email from one imap provider to another has proved a little trickier than expected.

The low-tech solution is to set up Thunderbird to check both accounts and drag messages from one to the other. Unfortunately you cannot create folders on dragging which makes it necessary to manually create all target folders. Moreover, if you interupt the copy process (for example by opening another folder) then it is aborted and not all messages will be copied across. This grows old very quickly.

Enter imapsync to the rescue.

Syncing the two accounts over ssl is as simple as:

imapsync --host1 mail.mailserver1.com --user1 username1 --password1 secret1 --host2 mail.mailserver2.com --user2 username 2 --password2 secret2 --ssl1 --ssl2


All necessary folders are created in the second account and flags and datestamps are preserved. The man pages are very comprehensive and deserve close reading.

Debian Etch contains the package by default and apt-get will handle all the perl dependencies. (Etch also includes a similar tool imapcopy which looks promising but I have yet to try it.) Users of other distros - or operating systems - should try the imapsync project page.

Edit - I did run into a problem with messages being copied more than once on subsequent reruns of imapsync.

This was easily solved by following the advice in the FAQ. Some IMAP servers add headers for each message transfered so each time you run the process, imapsync thinks all the messages are new. Rather than dig through the headers to work out exactly where this was happening, I took the lazy option and instructed imapsync to just use the Message-ID header when comparing messages and to ignore any size changes.

imapsync ... --useheader 'Message-ID' --skipsize

Voila. Four and half years worth of email effortlessly transfered from one IMAP account to another.

The google personalised homepage is back up with a new feature. The search giant has added themes to the pages, a charming series of bright, cartoonish skins that change depending on the time or weather.

To enable themes, you need to have modified your page already otherwise the option is not triggered. The impatient might consider adding random widgets until the option “Select Theme” appears to the left of “Add Stuff” at the top right of the page.

Mac users may have difficulty viewing themes (and/or the option to view themes) on their default home page, http://www.google.com/ig.

If you are using a Mac - Safari or Firefox - and themes are not showing on the google personalised home page - you should instead try the following url: http://www.google.com/ig?hl=en.

Again, if the option is not there yet, try adding some widgets and adjusting their settings (you can always delete them later).

Nicholas Negroponte coined the phrase “the Daily Me” in the mid-nineties to describe a digital newspaper personalised to the individual reader. Google ig is that newspaper but it’s not been working for the last few hours.

My configuration appears to have been (I assume temporarily) lost. I am missing it already.

David MacIver drew my attention to Chickenfoot today: “a Firefox extension that puts a programming environment in the browser’s sidebar so you can write scripts to manipulate web pages and automate web browsing.” It sounds like a slightly better Greasemonkey (although I must admit I haven’t touched Greasemonkey for a long time and it has no doubt changed since then.)

Being unable to install it was the final straw in making me decide to upgrade firefox. I’ve been running the default debian version out of a combination of laziness and paranoia about security but in the last few months I’ve really began to notice how out of date it is. More and more sites have been crashing my browswer. Fewer and fewer extensions have been working. But like the proverbial frog in the pan of boiling water, I have noticed all this happening but failed to jump out.

Turns out it was very simple to do. I just added backports my /etc/apt/sources.list

deb http://www.backports.org/debian sarge-backports main

then

apt-get update && apt-get install firefox -t sarge-backports

Now I’m finally able to install Chickenfoot but I haven’t played with it yet because I’m catching up on the 1000+ unread posts in bloglines that I haven’t been able to read for months since the sites last upgrade made it crash my browser.

Starting with Bjarne Stroustrup’s observation “our civilization runs on software”, Grady Booch offered a thought provoking overview of the history and promise of software at this year’s BCS Turing lecture, taking listeners from the austere beauty of Alan Turing’s 1930s thought experiments through to “the rise of the machines” in 2030.

Booch is an interesting, relaxed and witty speaker, whose asides on the superiority of OS X to Windows, George Bush and Google (”Am I the only one who thinks there’s a company in desperate need of some adult supervision?”) provided comic relief in an at times informationally dense speech.

One point that intrigued me was his observation that much of the history of computing is unrecorded, existing only in the “tribal memory” of the greybeards. He foresees the emergence of both software artists and historians who might translate and record some of the strange beauty of code for non-programmers as well as formally archiving a form of communication in danger of vanishing with the death of its authors.

The full lecture is available as a recording from the link below and is well worth watching.

Over 90% of email is now junk mail according to a recent BCS article.

Not only are spam volumes increasing, but junk messages are proving harder to filter. Some spam buries its message in images to prevent filtering on content. To reduce the number of these in your inbox, quarantine messages with headers containing “Content-Type: multipart/related” but check your quarantine folder regularly for false positives.

Others send spam for you to a third party and forge the sender’s address so that it appears to have come from you. This technique is known as backscatter. If you keep receiving messages with the header “Undelivered Mail return to sender” about emails you never sent then you may be the victim of this technique. Quaranting messages containing “Action: failed”, “Delivery Status Notification (Failure)” and/or with the subject containing “Undeliverable” will help reduce the volume of these although again this risks filtering off genuine messages alerting you to a failed delivery.

I’ve just started testing greylisting on my email. Email from unknown senders is temporarily bounced back: legitimate mail clients will try again later; spammers either will not try again or hopefully will have been added to a blacklist by the time they do so. Known senders are added to a whitelist and automatically bypass the greylist filter.

Mailsnare offer server-side greylisting. However, I have been disappointed with their service levels recently and am not sure how strongly I would recommend them.

I’ve noticed a plague of “Undelivered Mail return to sender” messages recently in my inbox - all of them spam. Friends with their own domain names have reported the same.

Clearly conventional spam filters are working too well and the vermin who make a living out of unsolicited junk mail are changing their tactics.

Now instead of spamming the victim direct, they forge the headers of their messages so that it appears to come from the person they wish to spam, send it to a non-existent account elsewhere and let the second account bounce it back to their real target.

This level of indirection helps it get through the victim’s spam filters.

If you have received email pitches about dubious stocks that are “about to take off” from this domain please rest assured that they do not originate with me.

Edit: For advice on tackling this problem, please read the following post on using greylisting.

I’ve just deleted a relatively innocuous comment on a recent post because, although it was not in itself spam, the poster gave his url as a site that stinks like a link-farm.

The IP address of the poster was from an Indian ISP which makes me wonder if some comment spammers frustrated with the efficiency of blocking tools are now abandoning automated scripts and outsourcing their work to low-waged workers. Of course, once a comment has been approved from a certain IP, it increases the chances that further comments will be approved so perhaps they are merely using people to get a foot in the door before launching an automated assault.

Either way, if your comment was genuine and I deleted it over-zealously I apologise. Somehow however, I doubt this was the case.