howto « Tim Hardy

Archive for the ‘howto’ Category

Editing and Removing Pages from PDF Documents

Sunday, January 11th, 2009

I’ve been using Google Docs to store and edit my CV, however I’ve recently run into a number of problems when exporting the file.

Word documents are invariably mangled. This is a common problem because Word format is a de facto standard by virtue of the number of companies using Microsoft Word but formatting varies from version to version, platform to platform so it’s not really a standard at all. It’s more frustrating than web development at times. You spend hours laying out your CV on a Mac only to find that it looks like hell when opened and printed on a PC.

Recruiters and employers that use text-processing algorithms to assess candidates for positions hate PDF documents but for anyone who cares about presentation and wants to guarantee that their potential employers see their CV exactly as they intended, there is no other choice.

Unfortunately, Google Docs insists on adding a line feed to the last line of every document and when you export the file as a PDF this can result in a blank page being appended to the end.

Fortunately you can edit PDFs on linux using pdftk.

For example, to create a new two page PDF from the first two pages of an original try:

pdftk originalCV.pdf cat 1-2 output editedCV.pdf

The application enables many more useful ways to manipulate PDF documents. Read the man pages for further details.

Posted in General IT, Linux, howto | Comments Off

Connecting to Ubuntu from iBook G4 Using NxMachine

Saturday, January 3rd, 2009

I’ve been using my girlfriend’s iBook recently and am very impressed by it.

The Good
Things I love include the fact that it feels like unix, the build quality of the hardware itself and the failsafe reliability of its sleep/resume.

(I’ve long forgotten the number of hours I spent a couple of years back disassembling then recompiling the buggy DSDT on my old IBM Thinkpad T20 to fix all the warnings and errors before linking it against a patched kernel in order to get ACPI working. Sure it gave me a taste of the days “when men were men and wrote their own device drivers” but sometimes it’s nice when things Just Work.)

The Bad
Things that niggle include the lack of right-mouse button, the unfamiliar keyboard layout and the absence anywhere on the keyboard of a hash/pound key which makes writing bash scripts a little tricky (it’s ALT+3 but for some reason this isn’t printed on the key itself).

The Ugly
Things that seriously annoy include the monolithic, closed nature of the operating system that requires you to upgrade the whole damn thing in order to use a more recent version of Java.

Early observations aside, connecting to my Ubuntu box using NxMachine was pretty straightforward.

The mac client is straightforward to download and install. Apt-get makes setting up the server on the linux box utterly painless. The instructions on the site are more than sufficient for getting the connection up and running.

Getting the key mapping right takes a little longer – out of the box several keys did not behave as expected.

Anyone looking to save a little time is welcome to use my keyboard settings. To apply them use xmodmap:

xmodmap keyboardsettings

Posted in General IT, Linux, howto | Comments Off

Retrieving Rapidshare Files with Python

Friday, January 2nd, 2009

A cursory google search will reveal several scripts for retrieving rapidshare files using python, but each one I’ve seen delegates the actual retrieval to wget.

This is not necessary.

Rapidshare uses basic authentication to identify logged in members and urllib2 can handle this easily.

The following method would do the trick without the need to call external executables:

def rapidget(url. login, password):
    "Retrieve files from rapidshare using only python"
    request = urllib2.Request(url)
    base64string = base64.encodestring('%s:%s' % (login, password))[:-1]
    request.add_header("Authorization", "Basic %s" % base64string)
    i = url.rfind('/')
    filename = url[i+1:]
    print url, "->", filename
    file = open(filename, 'wb')
    handle = urllib2.urlopen(request)
    buffer = ''
    buffersize = 1024*1024
    while True:
        buffer = handle.read(buffersize)
        if not buffer:
            handle.close()
            file.close()
            break
        file.write(buffer)
        buffer = ''
        print '.',

This assumes, of course, that you have an account at rapidshare.

Posted in Internet, Programming, Uncategorized, howto, python | Comments Off

Deleting Messages from an IMAP Folder Using Python

Saturday, August 16th, 2008

I was asked to help delete 16,980 messages from an IMAP spam folder the other day. No email client could handle it without crashing. Even mutt choked after several hours of valiantly struggling.

Python to the rescue. Rather than write a script to do this I ran each command from the python shell. It’s a very addictive way of working because you get instant feedback.
import getpass, imaplib M = imaplib.IMAP4_SSL("imap.gmail.com") M.login("yourusername@gmail.com", getpass.getpass())

Now we’re in. Let’s see what directories exist.
M.list()

Pick the offending directory.
M.select("[Gmail]/Spam")

View the messages.
typ, data = M.search(None, 'ALL') for num in data[0].split(): ....typ, data = M.fetch(num, '(RFC822)') ....print 'Message %s\n%s\n' % (num, data[0][1])

Now delete them all and close the connection to the mailserver. To delete a message in IMAP, you need to set the delete flag on it then expunge the folder.
typ, data = M.search(None, 'ALL') for num in data[0].split(): ....M.store(num, '+FLAGS', '\\Deleted') M.expunge() M.close() M.logout()

(To be fair to google I must point out that gmail was not the offending mailserver although I’ve used them in the sample code above.)

Posted in Internet, Programming, howto, python | 5 Comments »

Patching K3B on Kubuntu Hardy Heron

Tuesday, May 6th, 2008

I’ve recently built a new desktop machine – the subject of another post when I get time – and because the hardware is fairly recent I decided to try Ubuntu which I knew would support it rather than switch to an unstable branch of Debian.

My first impressions of Hardy Heron – Ubuntu’s latest release; I’m using the variant with the KDE desktop – are very positive. The installer is excellent and the hardware detection very impressive. As a comparison, it took me the better part of a day to install a fairly vanilla Windows XP Pro SP2 on the first partition of the box. Nearly every driver install required a reboot as did many of the security updates. By contrast, it took less than an hour to download and install Kubuntu and the installation includes a far greater range of packages than I have on the windows partition.

The one fly in the ointment has been a problem with K3b. If I started with a blank CD in the drive, all was fine but launching it with either an empty drive or a blank DVD caused the process to hang for a long while and finally launch with the warning:

“No CD/DVD writer found.
K3b did not find an optical writing device in your system. Thus, you will not be able to burn CDs or DVDs. However, you can still use other K3b features like audio track extraction or audio transcoding or ISO9660 image creation.”

After this the DVD drive was disabled, making it impossible to even eject until the machine had been shutdown completely and restarted.

Both growisofs and wodim worked from the command line so the problem had to be in the KDE application. Launching K3B from the console reveals debug information and showed the hang up was occurring after the step checking for RAW_R96R:

...
(K3bDevice::Device) /dev/scd0: checking for RAW_R96R

After a long pause the process would continue:

(K3bDevice::ScsiCommand) failed:
                           command:    GET PERFORMANCE (ac)
                           errorcode:  72
                           sense key:  NO SENSE (2)
                           asc:        0
                           ascq:       3
(K3bDevice::Device) /dev/scd0: GET PERFORMANCE length det failed.
(K3bDevice::ScsiCommand) failed:
                           command:    MODE SENSE (5a)
                           errorcode:  0
                           sense key:  NO SENSE (2)
                           asc:        0
                           ascq:       0
(K3bDevice::Device) /dev/scd0: MODE SENSE length det failed.
(K3bDevice::ScsiCommand) failed:
                           command:    MODE SENSE (5a)
                           errorcode:  0
                           sense key:  NO SENSE (2)
                           asc:        0
                           ascq:       0
(K3bDevice::Device) /dev/scd0: MODE SENSE with real length 65535 failed.
(K3bDevice::Device) could not open device /dev/scd0 for reading
                    (No such device or address)
Error while retrieving capabilities.
Could not initialize device /dev/scd0
Could not resolve /dev/scd0
/dev/scd0 resolved to /dev/scd0
(K3bDevice::Device) could not open device /dev/scd0 for reading
                    (No such file or directory)
could not open device /dev/scd0 (No such file or directory)
Devices:
------------------------------
kdecore (KAction): WARNING: KActionCollection::operator+=(): function is severely deprecated.
removing udi /org/freedesktop/Hal/devices/pci_8086_2922_scsi_host_0_scsi_device_                                                                                                                                                 lun0_scsi_generic
removing udi /org/freedesktop/Hal/devices/pci_8086_2922_scsi_host_0_scsi_device_                                                                                                                                                 lun0
removing udi /org/freedesktop/Hal/devices/pci_8086_2922_scsi_host_0
removing udi /org/freedesktop/Hal/devices/storage_model_DVDRW_LH_20A1S
Unmapping udi /org/freedesktop/Hal/devices/storage_model_DVDRW_LH_20A1S from dev                                                                                                                                                 ice /dev/scd0

dmesg reveals the following occuring

May  1 23:52:15 mingus kernel: [ 3580.143800]          cdb ac 00 00 00 00 00 00 00  00 01 03 00 00 00 00 00
May  1 23:52:15 mingus kernel: [ 3580.143801]          res 40/00:03:00:00:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
May  1 23:52:15 mingus kernel: [ 3580.455532] ata6: soft resetting link
May  1 23:52:15 mingus kernel: [ 3580.619411] ata6: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
May  1 23:52:45 mingus kernel: [ 3610.751976] ata6.00: qc timeout (cmd 0xa1)
May  1 23:52:45 mingus kernel: [ 3610.751985] ata6.00: failed to IDENTIFY (I/O error, err_mask=0x5)
May  1 23:52:45 mingus kernel: [ 3610.751993] ata6: failed to recover some devices, retrying in 5 secs
May  1 23:52:50 mingus kernel: [ 3615.752086] ata6: hard resetting link
May  1 23:52:56 mingus kernel: [ 3621.263794] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:53:00 mingus kernel: [ 3625.752312] ata6: hard resetting link
May  1 23:53:06 mingus kernel: [ 3631.268013] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:53:11 mingus kernel: [ 3635.804490] ata6: hard resetting link
May  1 23:53:16 mingus kernel: [ 3641.316197] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:53:46 mingus kernel: [ 3670.813264] ata6: limiting SATA link speed to 1.5 Gbps
May  1 23:53:46 mingus kernel: [ 3670.813266] ata6: hard resetting link
May  1 23:53:51 mingus kernel: [ 3675.821377] ata6.00: disabled
May  1 23:53:51 mingus kernel: [ 3675.821404] ata6: hard resetting link
May  1 23:53:57 mingus kernel: [ 3681.736761] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:54:01 mingus kernel: [ 3685.825584] ata6: hard resetting link
May  1 23:52:15 mingus kernel: [ 3580.143800]          cdb ac 00 00 00 00 00 00 00  00 01 03 00 00 00 00 00
May  1 23:52:15 mingus kernel: [ 3580.143801]          res 40/00:03:00:00:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
May  1 23:52:15 mingus kernel: [ 3580.455532] ata6: soft resetting link
May  1 23:52:15 mingus kernel: [ 3580.619411] ata6: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
May  1 23:52:45 mingus kernel: [ 3610.751976] ata6.00: qc timeout (cmd 0xa1)
May  1 23:52:45 mingus kernel: [ 3610.751985] ata6.00: failed to IDENTIFY (I/O error, err_mask=0x5)
May  1 23:52:45 mingus kernel: [ 3610.751993] ata6: failed to recover some devices, retrying in 5 secs
May  1 23:52:50 mingus kernel: [ 3615.752086] ata6: hard resetting link
May  1 23:52:56 mingus kernel: [ 3621.263794] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:53:00 mingus kernel: [ 3625.752312] ata6: hard resetting link
May  1 23:53:06 mingus kernel: [ 3631.268013] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:53:11 mingus kernel: [ 3635.804490] ata6: hard resetting link
May  1 23:53:16 mingus kernel: [ 3641.316197] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:53:46 mingus kernel: [ 3670.813264] ata6: limiting SATA link speed to 1.5 Gbps
May  1 23:53:46 mingus kernel: [ 3670.813266] ata6: hard resetting link
May  1 23:53:51 mingus kernel: [ 3675.821377] ata6.00: disabled
May  1 23:53:51 mingus kernel: [ 3675.821404] ata6: hard resetting link
May  1 23:53:57 mingus kernel: [ 3681.736761] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:54:01 mingus kernel: [ 3685.825584] ata6: hard resetting link
May  1 23:54:07 mingus kernel: [ 3691.752971] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:54:11 mingus kernel: [ 3695.841797] ata6: hard resetting link
May  1 23:54:17 mingus kernel: [ 3701.757192] ata6: port is slow to respond, please be patient (Status 0x80)
May  1 23:54:46 mingus kernel: [ 3730.858562] ata6: limiting SATA link speed to 1.5 Gbps
May  1 23:54:46 mingus kernel: [ 3730.858565] ata6: hard resetting link
May  1 23:54:51 mingus kernel: [ 3735.884188] ata6: EH complete
May  1 23:54:51 mingus kernel: [ 3735.884287] ata6.00: detaching (SCSI 5:0:0:0)

It looks like the bug is probably in the libsata code failing to handle an error rather than K3B but I haven’t had a chance to investigate further.

As a quick workaround, download the k3b source and hack bool K3bDevice::Device::getSupportedWriteSpeedsViaGP to return false immediately and shortcircuit the call to ioctl that locks the device. The code will fall back to setting a maximum speed determined by the previous call to getSupportedWriteSpeedsVia2A.

Posted in Linux, Programming, howto | 3 Comments »

Using Mutt with Exim4 to Send Mail via Smarthost over SSL (TLS) Connection on Port 465

Monday, November 26th, 2007

One day I’ll learn pine and I’m sure I’ll love it but for now I’m hooked on mutt.

Getting mutt to use exim4 for delivery is as simple as adding set sendmail="/usr/sbin/exim4" to your ~/.muttrc. Getting exim4 working as a smarthost to authenticate over port 465 is far harder.

The solution is to use a tunnel as outlined in this excellent post: exim4 with ssmtp on Debian.

Before finding this answer, I also discovered a very clear post on testing an smtp auth connection here and a useful tutorial on setting up and testing exim, both of which may be helpful to other people.

Posted in Internet, Linux, Uncategorized, howto | Comments Off

Debian Tip: How to Backup Installed Package List Before Reinstall/Upgrade

Sunday, October 14th, 2007

Many thanks to Renato for the following useful tip:

On your existing box:
COLUMNS=180 dpkg -l | awk '{print $2}' | xargs > /tmp/t

Then (having backed up then restored the list to /tmp/t, on the new install:
apt-get install `cat /tmp/t`

Posted in Linux, howto | Comments Off

Transfering the Nostalgia Ringtone to the N95

Saturday, September 8th, 2007

Following recommendations from Sumesh, Costas and Joyce, I’ve upgrade to the Nokia N95. It’s a great phone and the wireless is fantastic but it does have a few teething problems.

Once annoyance is the lack of a “normal” ringtone – or indeed anything that doesn’t make you look like a tasteless idiot when your phone goes off in pubic.

My favourite ringtone on the N70 was called Nostalgia. It sounds like an old fashioned, British telephone. Unfortunately the N95 does not come with this option.

Thankfully it’s easy to transfer from my old handset by using FileExplorer to navigate to Z:\Noka\Sounds\Digital\ then sending Nostalgia.aac to my new handset via bluetooth. Once saved to the new phone, it can be selected as a ringtone in Profiles.

Posted in Nokia, Symbian, howto | 1 Comment »

Solution to Problem Transfering Data from the Nokia N70 to the N95

Saturday, September 8th, 2007

There’s a bug with the Nokia data transfer application that stops it from working out of the box.

My old phone is a Nokia N70 but this should work for all other models.

If you set up your new phone and follow the clear instructions (that I won’t repeat here) then you will get as far as the N95 transferring an application called “DataMover.sis” to your old phone but no further. Trying to install DataMover.sis will give the warning: “Certificate may not yet be valid, is expired or phone’s date setting may be incorrect.”

The certificate has been incorrectly set to expire on 12 June 2007. To work around this, temporarily set your old phone’s date to some point before that in order to trick it into installing the application. (Tip: just change the year to 2006, it makes it easier to restore the correct date later.)

Once the application has been installed to your old handset the rest of the process should proceed smoothly.

Posted in Nokia, Symbian, howto | 4 Comments »

Migrating Emails from One IMAP Provider to Another

Sunday, May 20th, 2007

I have finally tired of mailsnare’s poor customer service. For a long while, the only way to get a response to a ticket was to post on the forums and publicly shame them into responding. Then they deleted the forums.

After trying several candidates, I have finally settled on webmail.us.

There are IMAP providers out there offering more bells and whistles but webmail.us advertises 99.99% uptime and offers 24 hour support by email, web or phone. With that kind of service, I can live without cutting edge, alpha-geek features that aren’t much use when the email server has been down for three days and no one is responding to requests for service…

Migrating several years worth of email from one imap provider to another has proved a little trickier than expected.

The low-tech solution is to set up Thunderbird to check both accounts and drag messages from one to the other. Unfortunately you cannot create folders on dragging which makes it necessary to manually create all target folders. Moreover, if you interupt the copy process (for example by opening another folder) then it is aborted and not all messages will be copied across. This grows old very quickly.

Enter imapsync to the rescue.

Syncing the two accounts over ssl is as simple as:

imapsync --host1 mail.mailserver1.com --user1 username1 --password1 secret1 --host2 mail.mailserver2.com --user2 username 2 --password2 secret2 --ssl1 --ssl2

All necessary folders are created in the second account and flags and datestamps are preserved. The man pages are very comprehensive and deserve close reading.

Debian Etch contains the package by default and apt-get will handle all the perl dependencies. (Etch also includes a similar tool imapcopy which looks promising but I have yet to try it.) Users of other distros – or operating systems – should try the imapsync project page.

Edit – I did run into a problem with messages being copied more than once on subsequent reruns of imapsync.

This was easily solved by following the advice in the FAQ. Some IMAP servers add headers for each message transfered so each time you run the process, imapsync thinks all the messages are new. Rather than dig through the headers to work out exactly where this was happening, I took the lazy option and instructed imapsync to just use the Message-ID header when comparing messages and to ignore any size changes.

imapsync ... --useheader 'Message-ID' --skipsize

Voila. Four and half years worth of email effortlessly transfered from one IMAP account to another.

Posted in Internet, Linux, howto | Comments Off

Tim Hardy