elliot's blog

Talis Library Platform News

I'm the featured team member in this month's Talis Library Platform News.

My favourite thing I wrote in 2007

I do a lot of computer programming, watching DVDs and so forth. But I have a lot of passion for writing. I write a lot of poetry. I know it's not fashionable, but I still do it, whenever the mood takes me. Last year I tried to write for at least 5 minutes every day, with varying success: I eventually stopped doing that, but did carry on writing, sometimes poring over a single poem for two or three weeks while travelling to work on the bus. Occasionally I wrote something good.

This is what I consider one of the best things I wrote last year. I did it on holiday in Wales, sitting in a car park while Nicola and Madeleine were going to the toilet or something. They were away for about 20 minutes, and I put together various bits of imagery from the car park and the rest of the holiday. I then wrote it up a few months later. I like the result. I don't think it's too pretentious (you might disagree), though on reflection it is a bit overwritten; though I think it genuinely has a fair few good lines. Here it is.

Welsh car park 2007-10-28

Queen Mab complains to crows about the gulls;
hobgoblins lunge from mossy limbs
with knotted arms, heads like balls of twine

the brow of the landscape frowns;
pleats of forest shake down the crag's skirt,
a mountain's shoulder shrugs above an elbow lake

as secret waterfalls shudder pebbles;
starlings cascade from the oak like gypsy pegs
round caravan steps; a row of recycling bins -

helmets of buried faery kings,
or giants' trepanned skulls; slapping feet
on the caravan roof, the curved loop of the seagull's call

a smell of dust in a vast car park
tumbles into thin scent of dog roses, stroked flat
by rising wind; a fringe of cloud like an anemone mauls

the peaks; crab skeleton, flesh scooped out;
our skeletons, green with algae, in the estuary flats -
we drive away too fast, and leave these things behind

Doing all sorts of burning, ripping and encoding of video and DVDs and audio (on Linux)

Using Linux every day means that I often grapple with how to re-encode proprietary formats so that I can watch them on the computer of my choice. I also do some DVD ripping and creating new DVDs of home movies etc., for which the Linux command line tools work very nicely (more quickly, more consistently and in a more stable fashion than some of the GUIs).

So I've gathered a whole load of tips on encoding, ripping, burning, culled from dozens of forums, websites, manpages etc.. This, then, is the current state of my understanding on this topic, and hopefully distills many hours of pain into an easily-digestible format. It's not very well organised, but hopefully useful. I should mention that this stuff works on Ubuntu, but your mileage may vary. Here goes.

A note on tools

All of the tools I use are easily installable on Ubuntu, either from official repositories or universe/multiverse. You will also need to install the proprietary codecs if you want to work with them. Here's what I tend to use:

  • vlc media player
  • mplayer media player
  • ffmpeg format transformer
  • mencoder
  • dvgrab
  • dvdauthor
  • dvdbackup
  • growisofs
  • mkisofs
  • sox
  • lame
  • cdrecord

Burning an ISO onto a DVD/CD

To work out where your CD/DVD device is:

$ cdrecord -scanbus
        1,0,0   100) 'HL-DT-ST' 'DVD+-RW GSA-T11N' 'A103' Removable CD-ROM
        1,1,0   101) *
        1,2,0   102) *
        1,3,0   103) *
        1,4,0   104) *
        1,5,0   105) *
        1,6,0   106) *
        1,7,0   107) *

What you're looking for is an entry with something other than a * in it; then get the content of the first column, which is the device ID (here, it's 1,0,0).


cdrecord -v -dao dev=1,0,0 image.iso

setting dev= to the ID you found above, and where image.iso is the path to your ISO image file.

Getting the audio out of a YouTube video into an mp3 file

First off, download the FLV version of the video. This is the tricky bit, but here's how to find it:

  1. Go to the YouTube page for the video
  2. Copy the content of the v parameter in the querystring (the video ID)
  3. View the source of the web page
  4. Search for "f": in the source; following this will be a long text string (the t parameter), e.g. "OEgsToPDskLJFcLZay_H2RO2hijVCxkP". This is some kind of secret token you'll need to do the download.
  5. Construct the URL as follows:
http://www.youtube.com/get_video?video_id=<video ID>&t=<t parameter>

Use your favourite browser or downloader to fetch the file (FLV format).

My new technique for getting the mp3:

ffmpeg -vn -i youtubevideo.flv youtubevideo.mp3

You'll need a recent ffmpeg for some YouTube videos, and will also need mp3 support compiled in to do this conversion.

My old technique for getting the mp3:

Once you've downloaded it, play it through mplayer, resampling the audio at 44.1KHz:

mplayer -vo null -vc null -ao pcm:file=out.wav -af resample=44100:0:0 youtubevideo.flv

Then, use lame to encode the wav file to mp3:

lame -h -V0 out.wav out.mp3

This does a reasonable quality, variable bit-rate mp3.

There are a couple of services which do this (e.g. YouTubeHack) and a command line script, but I couldn't get the services to work, and couldn't be bothered with the command line script.

mp4 or mov to mpg

Use mencoder for this. mp4 version:

mencoder in.mp4 -ovc lavc -oac lavc -o out.mpg

mov version is practically the same:

mencoder in.mov -ovc lavc -oac lavc -o out.mpg

Ripping from a dv camera with dvgrab

dvgrab --autosplit --timestamp --format jpeg

(The camera should be detected automatically, and I think this waits until it detects some input before the capture starts. I use a firewire cable to connect from the camera to the computer.) The files get named after timestamps coming from the film.

Encoding from .avi to .mpg


ffmpeg -i infile.avi outfile.mpg

(though this produces quite low quality output)

This produces better quality (bitrate = 800):

ffmpeg -i infile.avi -vcodec mpeg2video -acodec mp2 -b 800 outfile.mpeg

Encoding from .avi to .mpg suitable for DVD burning

ffmpeg -i finalmovie.avi -y -target pal-dvd -sameq -aspect 16:9 finalmovie.mpg

If you're in the US, changed pal-dvd to ntsc-dvd.

Ripping DVDs to hard disk

(See http://www.bunkus.org/dvdripping4linux/single/index.html for lots of good tips)

May need dvdcss decoder to be installed:

sudo /usr/share/doc/libdvdread3/examples/install-css.sh

Use dvdbackup to rip a DVD to hard disk.
See http://dvd-create.sourceforge.net/dvdbackup-readme.html for full instructions

# Get info. about DVD
dvdbackup -i /dev/cdrom -I

# Rip whole DVD
dvdbackup -M -i/dev/cdrom -o /media/usbdisk/dvdripping

# Rip main feature
dvdbackup -F -i /dev/cdrom -o /media/usbdisk/dvdripping

# Rip title set (in this case, title set 2)
dvdbackup -T 2 -i /dev/cdrom -o /media/usbdisk/dvdripping

# Rip title (here, rip title 1)
dvdbackup -t 1 -i /dev/cdrom -o /media/usbdisk/dvdripping

Direct DVD copying

This effectively copies the DVD's iso image to hard disk:

dd if=/dev/cdrom of=file.iso bs=2048

If the disk is encrypted, this might fail. In this case, it might be worth running this first:

dvdbackup -I -i /dev/cdrom

Then try dd again.

See http://gentoo-wiki.com/HOWTO_Backup_a_DVD#dd for more details

Note that this produces a mountable DVD image. However, it does not remove encryption, so you would still need to rip to individual VOBs using dvdbackup to get rid of that. However, it is possible to mount the iso and play it as if it were a DVD (see later): vlc is good for this.

Playing a partially ripped DVD

If you've ripped some of the content of a DVD (e.g. using dvdauthor -i /dev/cdrom -F), you can play the partial rip with:

mplayer dvd:// -dvd-device <ripped_dvd_directory>

(vlc might also be able to do this)

Encoding audio out of a VOB file (note that this drops the video altogether)

mplayer -vo null -vc null -ao pcm:file=outfile.wav infile.VOB

You could probably use ffmpeg for this, too.

Creating a DVD using dvdauthor when you don't want the whole disc

If you just have the main feature (a set of .VOB files) and you want to create a playable DVD from them.

Create a dvd.xml file in the top level directory of the ripped DVD (e.g. in VIDEO_TS)

  <vmgm />
          <vob file="VTS_01_1.VOB" />
          <vob file="VTS_01_2.VOB" />
          <vob file="VTS_01_3.VOB" />
          <vob file="VTS_01_4.VOB" />
          <vob file="VTS_01_5.VOB" />

(Adding another element for each VOB which you want included in your "movie". Note that these can equally well be your own mpg files.) You can also add titles etc.: see the man page for dvdauthor for more details. This works fine for me, though.

Then run it with:

dvdauthor -o <output directory for DVD structure> -x dvd.xml

Burning complete DVD structure to a new DVD

See http://www.linux.com/articles/53702

(I think you might need to have used dvdbackup -M (complete rip) for this to work, or have a DVD structure created using dvdauthor (see above))

growisofs -dvd-compat -Z /dev/cdrom -dvd-video <ripped dvd structure>

Creating an iso from a ripped DVD structure

If you've ripped the structure of a DVD (e.g. using dvdbackup) or created your own DVD structure (e.g. using dvdauthor), you can turn it into a single iso file with:

mkisofs -dvd-video <ripped dvd directory> | dd of=file.iso obs=32k seek=0

Mounting an iso filesystem so you can read it

mkdir mountpoint
sudo mount -o loop file.iso mountpoint

Once mounted, you can play the mounted iso (including its menu system) using vlc. For example, if we mounted it on the directory "mountpoint" we could do:

vlc dvd:///path/to/mountpoint

Encode VOB to mpg

If you ripped a VOB off a DVD but you want a smaller mpg:

ffmpeg -i VTS_01_1.VOB -vcodec mpeg2video -acodec mp2 -b 1000 sleeper1.mpg

Ripping RealPlayer streams

See https://help.ubuntu.com/community/HowToRipRealaudioStreamsToMp3 for full instructions

Short version:

vsound --timing --dspout --file=myfilename.wav realplay http://url.to.rip

Ruby script for converting from m4a (iTunes format?) to mp3

This is a shell script for converting files from m4a to mp3. It uses mplayer and lame behind the scenes. sox is supposed to do this, but I can never work out how to install proprietary codecs for it. This doesn't retain tags, unfortunately. Sneetchalizer will probably also work.

# only works if the script is in the directory with the m4a files
prefix = "new_file_prefix"
Dir['*.m4a'].each do |f|
  new_filename = prefix + File.basename(f).gsub(' ', '_').slice(3..-5).downcase
  wav = new_filename + ".wav"
  mp3 = new_filename + ".mp3"
  `mplayer -ao pcm:file='#{wav}' '#{f}'`
  `lame -h -b192 #{wav} #{mp3}`
  `rm #{wav}`

Ruby script for converting from ogg to mp3

Uses ogg123. This script loses all the tags, though. Sneetchalizer would do this, too.


filename = ARGV[0]

if filename.nil? or !(/\.ogg$/ =~ filename)
  puts "I don't think that's an ogg file, mister"

base_filename = filename.chomp(".ogg")
mp3_filename = base_filename + ".mp3"

`ogg123 -d au -f - #{filename} | lame - #{mp3_filename}`

mp3 to ogg

Bit basic, this, but you get the idea:

mpg321 -s input.mp3 | oggenc -r -a "artist" -t "title" -b 100 -M 140 -o output.ogg -

ogg to wav

sox is OK for this, as neither codec is proprietary:

sox input.ogg output.wav

BBC iPlayer Ruby code

I threw together some code for querying and parsing the BBC iPlayer search pages and emailing the results to you. You configure it by putting the names of the programmes you want to look out for into config.yaml, along with your email details, e.g.

  - mighty boosh
  - lead balloon
  - never better

  email_to: email@example.com
  email_from: email@example.com
  server: mail.example.com
  user: email@example.com
  pass: password
  auth_type: login

Copy the sample config.yaml.dist to config.yaml in the same directory and edit.

I run the command line script via cron once a day by calling the cli.rb script with an --email switch, e.g. with the following line in crontab:

0 21 * * * /usr/bin/ruby /home/ell/dev/iplayer/cli.rb --email

You could as easily run it from a Windows scheduled task.

Dependencies are:

  • hpricot
  • rack

What it does is request the iPlayer search page with each search term, one after the other. If there are multiple pages of results, it fetches each of those too, aggregating the results. It will then email you a list of links to the programmes on the iPlayer site. One thing it does which the iPlayer search page doesn't do is sort the matching results by how long is left for you to watch them: the ones with the least amount of time left are at the top.

You can also run it as a local web server on port 3334 with:

ruby server.rb

Which then becomes accessible at:


Nothing fancy: just an HTML page with the search results in it, using the same config. as the command-line client. You can also call the page with extra search parameters to perform custom one-off searches, e.g.


It's not a serious project, just a convenience for me. GPL licence.

Occurrences since Christmas

So, what's been happening?

  • Christmas - very nice, relaxing, though I was ill all the time. Ended up turning into sinusitis and me being on antibiotics. Got some nice DVDs for presents and a few books. In particular, I got 1001 Albums to Hear Before You Die. Though slightly disappointing (I already knew nearly all the bands in it, and whether I liked them), it did urge me to buy some new CDs with my Christmas money. I ended up buying:
    • Skylarking by XTC. Haven't listened to it yet.
    • The United States of America by The United States of America. This is a little gem of experimental 1960s electronic pop.
    • Inspiration Information by Shuggie Otis. This is a soul/funk/instrumental album from the 1960s, which includes early experiments with a drum machine, and has occasional moments of unhinged brilliance.
    • Reformation Post TLC by the Fall. This one's OK, but no stand-out tracks for me.
    • Station 55 by Cristian Vogel. Marvellous modern electronica. The first track is a marvel, including some excellent farting horns of impending doom. The third track is absolutely barmy acid house with some shrieking woman on top.
    • 2 EPs by the Aphex Twin trading as Tuss. Haven't got these yet.
      You can see the full list of my listening habits, as always, on Last.fm.
  • Nicola is pregnant! After having three miscarriages and both of us having some miserable months, she's now 17 weeks pregnant. Due date is June. I just hope I can remember what to do with babies. Madeleine is quite interested, and has suggested the name Emily, which we may use as a middle name if it's a girl.
  • We're moving. We've bought a new house, still in Northfield, and should be moving at the end of this month/start of next. It's bigger, with a bigger garden, near one of the parks. So I've been sorting out the mess in the current garden in readiness.
  • Work at Talis is going well. I feel like I've settled in a bit now, and am more at home. Since Christmas I've been working on RESTful APIs for bibliographic services. This has involved some Java programming, RDF and network programming, servlets etc.. Good fun. I've also been working on some open source library stuff under the banner of Jangle. Work is on hold at the moment and we're having a meeting tomorrow to discuss future directions. It's been very interesting, and one of my colleagues has got people from the Koha and Evergreen (the two main open source library systems) development teams interested in getting involved.
  • I spent one evening last week writing some Ruby code to parse the BBC iPlayer search pages. I've set it up to check for programmes I'm interested in watching on iPlayer, and it emails me a list of matching search items every day (run from cron). I also put a web server front end onto it (with the wonderful Rack) for fun. I will post it up here once I get round to it.
  • I've been watching oodles of Futurama and Firefly this week. Science fiction ahoy! Haven't read a book for ages, though.

acts_as_father_christmas plugin

Tired of your website not being at all festive? Rails applications lacking that Christmas zing? You need my new Rails plugin, acts_as_father_christmas, to bring that holiday spirit to your application.


  • Inserts random festive phrases into the text of your Rails application's views.
  • Optionally shows messages in even more intrusive yet festive RED CAPITAL LETTERS!
  • Can also add a wintry background image to your site to make users feel like Christmas is really coming.


Here's your dull, dry, non-festive site:

Now with acts_as_father_christmas installed:

Finally, with all the trimmings:


It's a standard Rails plugin, so from inside your app., do:

$ ruby script\plugin source http://svn.receptacular.org/plugins
$ ruby script\plugin install acts_as_father_christmas


To turn any controller into a Santa's grotto of delights, add this line inside a controller class definition:

class TestController < ApplicationController

To turn on red capital letters, pass the optional :merry => true:

class TestController < ApplicationController
  acts_as_father_christmas(:merry => true)

Finally, to sprinkle some magic over your site, pass in a block and call the magic method inside it:

class TestController < ApplicationController
  acts_as_father_christmas(:merry => true) do

Todo list

  • Inject dependent elves and reindeers, e.g. using Needle.
  • Add a santas_little_helper class to enable easier view customisation.
  • Provide options for circumventing ChimneySpace constraints, e.g. use of magic RSA key to open front door instead of using ChimneySpace pipes.
  • Build Santa on Sleighs web framework for Christmas-oriented content managed websites.
  • I could go on...


This is a fully-working Rails plugin, but if you're daft enough to use it and it breaks stuff, don't blame me. MIT Licence, in case you were wondering.

Sneetchalizer: nice Ruby audio format transformer/tag transferrer

I have quite a few ogg files, as well as a few mp3s. Occasionally I like to change formats around so I can play oggs on my mp3 player, or give them to friends and family. I have a few batch scripts lying around which do this using ogg2mp3, toolame etc., but they lose the ogg tags and I usually end up with mp3s without tags.

I thought things could be better, so I went in search of a tool to help. I found a Ruby script called The Sneetchalizer, which is great. To get it working I had to install ruby-mp3info (using gem) and download ruby-ogginfo. The latter isn't a gem, so you have to make sure you include it in Ruby's load path (see the example below). Also, rather than install it, I just pulled the sneetchalizer file out of the tarball, added a .rb to the end of it, and ran it directly using Ruby.

Sneetchalizer uses a whole bunch of Linux command line tools and Ruby libraries to do its work, but providing you've got them installed (probably you just need oggenc, oggdec and lame or toolame), it works a charm. It supports a whole load of audio formats and options, so had no problem with my ogg to mp3 challenge.

Here's an example of the command line I used to translate some ogg files in one directory into mp3s in another one:

ruby -I ~/Desktop ~/Desktop/sneetchalizer.rb --out=mp3 --in=ogg -r --sanitize \
-D /home/ell/no_backup/music/for_chloe_mp3s /home/ell/no_backup/music/for_chloe

Notice the -I option passed to Ruby, which makes any Ruby files in the stated path (~/Desktop in my case) available on the load path. I also like the fact that you can tell it to sanitize tags (--sanitize); and work recursively (-r).

AxleGrease rises from the dead

I've been fiddling with Merb lately (more another time), and found that on Linux the DataObjects drivers don't install very easily. To get them to work under the old version of AxleGrease (my pioneering :) Ruby on Rails on XAMPP distribution for Linux) was proving particularly traumatic. Plus I wanted to play with Rails 2, now it's out.

As a consequence, I decided to dust off and update AxleGrease for myself, and thought in the process I might as well put the new version up on RubyForge. Which I have. So you can get hold of it if you like. It has Merb but only supports DataObjects with SQLite3, not MySQL. I tried for about 2 hours to get MySQL support working, but couldn't. I did manage to compile the C code, but couldn't build a gem, as I think XAMPP is missing one of the MySQL libraries that DataObjects expects. I gave up (I use SQLite 3 for development anyway).

By the way, why Merb? Well, because I want to write a Ruby application which can easily connect to multiple database backends, and has decent support for a plugin-style architecture. Merb has both of those. Also, Merb is very slim and seems more suitable for writing frameworks; I've been trying to do this with Rails, but writing a framework inside a framework for writing apps. (if you follow me) is a bit of a headache at times. The only thing I can't do with Merb (yet) is run it under JRuby. But I don't think that will be far away.

Also, I am loving some of the new features in Rails: in particular, isolation of initializer code into its own directory (so environment.rb doesn't turn into a quagmire); resource routing and the whole ActiveResource thing; and the support for easy switching of output formats (respond_to). Unfortunately, I am using Rails Engines for the project I'm working on, and they don't work with Rails 2 yet. Although I imagine it can't be long.

95 pages of my Rails book for free!

Seems like Packt have decided to give away a fairly substantial chapter of my Rails book. You can get chapter 4 for free (PDF). This one is about Rails' architecture, MVC, database configuration and troubleshooting, migrations, validation and unit testing. Get it while it's free!

I could do with some feedback on it. I'm not sure if anyone's read it or likes it. I never considered this aspect of being a writer before: you have no idea of how well your book is selling, other than the vague indications given by Amazon. It's a bit odd really.

Ubuntu Feisty on Dell Latitude D820

I recently got a Dell Latitude D820 at work, and spent about 10 days pulling my hair out using Windows XP on it. Once I'd run out of hair, and after clearing it with the technical support people and my bosses, I persuaded them to let me install Linux (though they insisted on me keeping a small Windows boot partition).

Anyway, I've been using it successfully for the last 6 weeks or so, and it's been working great. Today, I worked from home for the first time, and wanted to get a few other things working:

  • I wanted to be able to move the machine around the house (wireless).
  • I wanted to get the touchpad working properly (i.e. scroll when I slide my finger down the side, and turn off taps being translated to mouse clicks).
  • It would be nice to get suspend to RAM working, so I can close the lid when I'm having my lunch and save some electricity.

In summary, I managed to get this lot working fine, and would say this machine is a nice one for running Linux. I don't like it quite as much as my old Thinkpad Z61t (keyboard isn't quite so nice), though I've never managed to get the wireless working on that.


The wireless worked fine, but for some reason the network applet just refused to allow me to connect. Eventually, I managed to get it working by hard-coding the IP address into the network config., and setting up the router to assign a static IP to the wireless card. But there was no need to install any drivers or anything: all of that was just working out of the box.


The touchpad was another matter, and took me about half an hour to figure out. In the end, after having read about 6 different articles on installing various shades of Linux on this model of laptop, I came across this extremely useful article, which enabled me to fix it definitively.

Eventually, I had to make two edits to /etc/X11/xorg.conf (as root). First, I added a line to the ServerLayout configuration element:

Section "ServerLayout"
        Identifier      "Default Layout"
        Screen "Default Screen"
        Inputdevice     "Generic Keyboard"
        Inputdevice     "Configured Mouse"
        # HERE'S MY EDIT
        Inputdevice     "Touchpad"      "AlwaysCore"
        Inputdevice     "stylus"        "SendCoreEvents"
        Inputdevice     "cursor"        "SendCoreEvents"
        Inputdevice     "eraser"        "SendCoreEvents"

Then I added a new section for the touchpad itself:

Section "InputDevice"
         Driver "synaptics"
         Identifier "Touchpad"
         Option "Device" "/dev/input/event3"
         Option "Protocol" "event"
         Option "LeftEdge" "130"
         Option "RightEdge" "840"
         Option "TopEdge" "130"
         Option "BottomEdge" "640"
         Option "FingerLow" "7"
         Option "FingerHigh" "8"
         Option "MaxTapTime" "0"
         Option "MaxTapMove" "0"
         Option "EmulateMidButtonTime" "75"
         Option "VertScrollDelta" "20"
         Option "HorizScrollDelta" "0"
         Option "UpDownScrolling" "1"
         Option "MinSpeed" "0.70"
         Option "MaxSpeed" "1.20"
         Option "AccelFactor" "0.080"
         Option "EdgeMotionMinSpeed" "200"
         Option "EdgeMotionMaxSpeed" "200"
         Option "SHMConfig" "on"
         Option "Emulate3Buttons" "on"

The important bit is where I specify the Device as /dev/input/event3. Also note I set MaxTapTime to 0 to turn off taps acting as mouse clicks. You may have to muck around with the *Speed and AccelFactor settings to suit you: there is a GUI client called gsynaptics which can configure scroll speed and taps; and there is a more comprehensive command line application synclient which helps if you want to fiddle around with the settings in real time.

Suspend to RAM

I found this article which provided the necessary settings. After I'd made the changes suggested, everything worked brilliantly: even the wireless came back from suspend. (Note that I use the proprietary NVidia drivers, so the instructions are specific to that.)

First off, I edited /etc/X11/xorg.conf, adding a couple of settings to the NVidia driver setup:

Section "Device"
        Identifier      "nVidia Corporation G72M [GeForce Go 7400]"
        Driver          "nvidia"
        Busid           "PCI:1:0:0"
        Option          "UseEDIDDpi"            "False"
        Option          "AddARGBVisuals"        "True"
        Option          "AddARGBGLXVisuals"     "True"
        Option          "NoLogo"                "True"
        Option          "NvAGP"                 "1"

Then I edited /etc/defaults/acpi-support (in all cases, the default setting was true):

Syndicate content