Home

howtos

warning: Creating default object from empty value in /home/townxorg/public_html/modules/taxonomy/taxonomy.module on line 1364.

Using wget to mirror a website

Wed, 2010-05-12 09:29 — elliot

Occasionally you need to mirror a website (or a directory inside one). If you've only got HTTP access, there are tools like httrack which are pretty good (albeit pretty ugly) at doing this. However, as far as I can tell, you can't use httrack on a password-protected website.

curl can probably do this too, and supports authentication, but it wasn't obvious.

So I ended up using wget, as it supports mirroring and credentials. But the issue here is that wget plays nice and respects robots.txt; which can actually prevent you mirroring a site you own. And nothing in the man page explains how to ignore robots.txt.

Eventually, I came up with this incantation, which works for me (access to password-protected site, full mirror, ignoring robots.txt):

wget -e robots=off --wait 1 -x --user=xxx --password=xxx -m -k http://domain.to.mirror/

where:

  • -e robots=off obviously disables robots
  • --wait 1 forces a pause between gets (so the site doesn't get hammered)
  • --user and --password: self-evident
  • -x creates a local directory structure which "mirrors" (see what I did there?) the directory structure on the site you're mirroring
  • -m turns on mirror mode: "turns on recursion and time-stamping, sets infinite recursion depth and keeps FTP directory listings" (from the man page)
  • -k converts links after download so that URLs in the mirrored files reference local files

Don't use it carelessly on someone else's website, as they might get angry...

Example of how to put RDFa into an HTML list

Tue, 2009-06-30 09:52 — elliot

I'm not going to try to explain RDF and/or RDFa here, but thought any poor suckers looking for RDFa examples might benefit from me posting what I finally worked out, with help from my colleague Rob. Namely, how to annotate an HTML ordered list (<ol>) with RDFa attributes; and how to put RDFa attributes onto form elements.

Here's the HTML page with RDFa embedded in it. What I'm representing here is a sequence of collections, and the individual collections within it:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML+RDFa 1.0//EN" "http://www.w3.org/MarkUp/DTD/xhtml-rdfa-1.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <title>Collections</title>
  </head>
  <body>
    <h1>Collections</h1>
    <form method="post" action="http://receptacular.org/collections">
      <ol xmlns="http://www.w3.org/1999/xhtml" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:xsd="http://www.w3.org/2001/XMLSchema#" xmlns:rec="http://receptacular.org/schema#" typeof="rdf:Seq" about="http://receptacular.org/collections">
        <li rel="rdf:_1" resource="http://receptacular.org/collections/1">
          <span style="display:none;" rel="rdf:type" resource="http://receptacular.org/schema#Collection"></span>
          <div class="collection-label" property="rdfs:label">Vague Collection</div>
          <input type="checkbox" id="collections-1-hidden" property="rec:hidden" datatype="xsd:boolean" content="false"/>
          <label for="collections-1-hidden">hidden</label>
          <input type="checkbox" id="collections-1-defaultSearch" property="rec:defaultSearch" datatype="xsd:boolean" content="false"/>
          <label for="collections-1-defaultSearch">use for searches</label>
        </li>
        <li rel="rdf:_2" resource="http://receptacular.org/collections/2">
          <span style="display:none;" rel="rdf:type" resource="http://receptacular.org/schema#Collection"></span>
          <div class="collection-label" property="rdfs:label">Archive Collection</div>
          <input type="checkbox" id="collections-2-hidden" property="rec:hidden" datatype="xsd:boolean" content="false"/>
          <label for="collections-2-hidden">hidden</label>
          <input type="checkbox" id="collections-2-defaultSearch" property="rec:defaultSearch" datatype="xsd:boolean" content="false"/>
          <label for="collections-2-defaultSearch">use for searches</label>
        </li>
        <li rel="rdf:_3" resource="http://receptacular.org/collections/3">
          <span style="display:none;" rel="rdf:type" resource="http://receptacular.org/schema#Collection"></span>
          <div class="collection-label" property="rdfs:label">Main Collection</div>
          <input type="checkbox" id="collections-3-hidden" property="rec:hidden" datatype="xsd:boolean" content="true" checked="checked"/>
          <label for="collections-3-hidden">hidden</label>
          <input type="checkbox" id="collections-3-defaultSearch" property="rec:defaultSearch" datatype="xsd:boolean" content="true" checked="checked"/>
          <label for="collections-3-defaultSearch">use for searches</label>
        </li>
      </ol>
      <p>
        <input type="button" value="Save" id="save-collections"/>
      </p>
    </form>
  </body>
</html>

Available online here: http://receptacular.org/collections

Things of note:

  • The doctype declaration. This is the W3C sanctioned doctype for XHTML+RDFa pages. By the way, the W3C validator will correctly validate this page, but standard XHTML validators don't (e.g. like "this one"http://nutrun.com/weblog/xhtmlvalidator-validate-xhtml-in-java/:). That's another story...
  • Namespace declarations on the <ol> element wrapping the list items. This is what causes standard XHTML validation approaches to fail.
  • The <ol> element is defined with typeof="rdf:Seq" and about="http://receptacular.org/collections". This sets it up as the RDF Seq resource.
  • The <li> elements inside the <ol> are Seq items, within the wrapping Seq resource. Each is each defined as a resource using the resource attribute. They are linked back to the enclosing <ol> element using the rel attribute on each. Note that the value for the rel attribute is an rdf Seq number, which orders the items within the enclosing Seq resource.
  • The RDF type of each Seq item is set using a hidden <span> element. Note that these elements have no text in them, but have opening and closing tags. If you just use a self-closing start tag for this element, the HTML doesn't display property.
  • The <span> elements use the rel attribute to mark their RDF type relationship to the outer list item; and the resource attribute to specify the location of the resource representing their type.
  • Each Seq item has an enclosed <div> which represents its rdfs:label property. Note that the property RDFa attribute is used to specify which property of the enclosing resource is being defined. Also note that the value of the property is inlined between the start and end tags of the <div>.
  • The two checkboxes define two more properties for each Seq resource: rec:hidden and rec:defaultSearch. (The semantics of the properties aren't discussed here, as I'm concentrating on syntax.) Each is defined on a standard XHTML <input> element: the relationship to the enclosing Seq item is defined with the property attribute; the value of the property is defined using the content attribute; and the data type of the literal value is defined via the datatype attribute. Any of the standard XML datatypes could be used here, or other types from other schemas.
  • When working with form elements which represent property values for RDF resources, you may need to change the content attribute in response to UI changes. (In the application from which this example was extracted, we use JQuery to respond to changes in the check box which set the content attribute.)

To see the RDF which can be extracted from this page, you can use the W3C's RDFa Distiller. Here's the resulting RDF:

<?xml version="1.0" encoding="utf-8"?>
<rdf:RDF
  xmlns:dist="http://www.w3.org/2007/08/pyRdfa/distiller#"
  xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
  xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
  xmlns:rec="http://receptacular.org/schema#"
  xmlns:xhv="http://www.w3.org/1999/xhtml/vocab#"
  xmlns:xml="http://www.w3.org/XML/1998/namespace"
  xmlns:xsd="http://www.w3.org/2001/XMLSchema#"
>
  <rdf:Seq rdf:about="http://receptacular.org/collections">
    <rdf:_1>
      <rec:Collection rdf:about="http://receptacular.org/collections/1">
        <rec:hidden rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">false</rec:hidden>
        <rec:defaultSearch rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">false</rec:defaultSearch>
        <rdfs:label>Vague Collection</rdfs:label>
      </rec:Collection>
    </rdf:_1>
    <rdf:_2>
      <rec:Collection rdf:about="http://receptacular.org/collections/2">
        <rec:hidden rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">false</rec:hidden>
        <rec:defaultSearch rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">false</rec:defaultSearch>
        <rdfs:label>Archive Collection</rdfs:label>
      </rec:Collection>
    </rdf:_2>
    <rdf:_3>
      <rec:Collection rdf:about="http://receptacular.org/collections/3">
        <rec:hidden rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">true</rec:hidden>
        <rec:defaultSearch rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">true</rec:defaultSearch>
        <rdfs:label>Main Collection</rdfs:label>
      </rec:Collection>
    </rdf:_3>
  </rdf:Seq>
</rdf:RDF>

Saving changes to an RDFa-enabled form like this is another challenge, for which we used rdfquery, and RDF library for JQuery. (I recommend you use the latest svn HEAD version of this library, as older versions have a bug where they ignore RDFa elements nested inside elements without RDFa attributes.) Maybe I'll get round to that another time.

Script to copy random mp3s from a directory to an mp3 player

Tue, 2009-06-30 09:24 — elliot

This is a Ruby script which randomly copies mp3 files from one directory to an mp3 player. I wrote it so I could easily fill up my mp3 player from the 9000 odd mp3s I have on a different external drive.

To run it, you'll need the sys-filesystem gem (see http://rubyforge.org/projects/sysutils):

$ gem install sys-filesystem

Next, edit these variables in the script (near the top):
* source_dir to the directory containing the mp3s you want to select from
* dest_dir to the path for the directory on your mp3 player you want to copy to

Be a bit careful, as this will attempt to fill the dest_dir you specify with mp3 files from source_dir. You might end up filling the wrong disk up.

Then just run it with ruby from the command line:

$ ruby mp3s_random.rb

Note that it won't delete anything from the destination drive, and will attempt to fill all the space available. Also note that it doesn't keep trying mp3s until it finds one which will fit the last remaining space: once it tries to copy a file which won't fit, it stops. You can always run it again to see whether the next run finds a file small enough to fit.

I've only tested it on Linux, but, who knows, it might work on Windows too. (No operating-system specific commands are used in the script, as it uses Ruby for all file operations.)

The code is below, but I've attached it as well.

require 'rubygems'
require 'sys/filesystem'
require 'ftools'

source_dir = '/media/disk/music'
dest_dir = '/media/disk-1/music'

files = Dir[File.join(source_dir, '/**/*.mp3')]
num_files = files.size

stat = Sys::Filesystem.stat(dest_dir)

disk_free_space_kb = (stat.blocks_free * stat.fragment_size).to_kb

files_selected = []

while disk_free_space_kb > 0 and num_files > 0 do
  # choose an mp3
  file_num_to_copy = rand(num_files - 1) + 1
  file_path = files.delete_at(file_num_to_copy)

  num_files = num_files - 1

  # work out how big file is
  file_size_kb = File.stat(file_path).size.to_kb

  # subtract from free space
  if (disk_free_space_kb - file_size_kb) > 0
    files_selected << file_path
    disk_free_space_kb = disk_free_space_kb - file_size_kb
  else
    break
  end
end

files_selected.each do |f|
  copy_to_path = File.join(dest_dir, File.basename(f))
  puts "Copying #{f} to device"
  File.copy(f, copy_to_path)
end

Decoding FLV files with ffmpeg

Tue, 2009-05-12 09:50 — elliot

I'm using Ubuntu Intrepid Ibex, but the ffmpeg distribution it comes with doesn't support recent FLV file encodings (like some videos from YouTube). You get an error like this when you try to do anything with them:

[flv @ 0xb800e4c8]Unsupported video codec (7)

My solution was to checkout ffmpeg from its Subversion repository and compile it myself:

$ svn checkout svn://svn.ffmpeg.org/ffmpeg/trunk ffmpeg-svn
$ cd ffmpeg-svn
$ ./configure --prefix=~/apps/ffmpeg-svn --enable-libmp3lame
$ make install

To do the compile, you'll probably need build-essential, as well as libmp3lame-dev, and the *-dev versions of any other codecs you want to use.

Use the resulting binary to do the conversion:

$ ~/apps/ffmpeg-svn/bin/ffmpeg -i infile.flv outfile.mpg

It turns out this version does support those recent FLV files, happily.

Describing (finding) subjects which don't have a particular predicate in SPARQL

Tue, 2009-04-07 09:37 — elliot

If you want to do something like a SQL NOT in SPARQL, here's what the query looks like:

PREFIX rs: <http://schemas.talis.com/2006/recordstore/schema#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

DESCRIBE ?tenancy {
  ?tenancy rdf:type rs:Tenancy .
  OPTIONAL { ?tenancy rs:platformStoreUri ?o } .
  FILTER ( !bound(?o) )
}

Here I'm looking for subjects with rdf type of http://schemas.talis.com/2006/recordstore/schema#Tenancy, which don't have a http://schemas.talis.com/2006/recordstore/schema#platformStoreUri predicate. The important bit is that you make the predicate which could potentially not be "set" OPTIONAL; and add a FILTER which only includes subjects where the predicate is bound to a value. This effectively screens out any subjects where the predicate has not been added to the subject. This pattern is basically Negation as Failure (according to the SPARQL recommendation), which derives from logic programming. Feels a bit like being back at university.

Installing Windows on the second hard disk of a Linux machine

Tue, 2009-03-31 22:18 — elliot

I recently upgraded the hardware of my old desktop PC, with the aim of providing the house with a new-ish Linux machine for watching movies and using the internet, and a Windows machine for writing music and playing (old) games. My plan was to use two hard disks: one for Linux, another for Windows, and choose which to use at boot time.

Normal procedure is to install Windows first, then install Linux into a spare partition on the same hard drive (Windows tends to overwrite any disk you put it on). But it's easier to get a Linux machine up and running, see what hardware you've got, and get a decent system without needing to go and find loads of old drivers. So I decided to install Linux first. I plugged in a drive for it as the Primary IDE drive, and installed Ubuntu Linux onto it.

Then, I unplugged the Linux drive, plugged the other drive in, and installed Windows 2000 onto the second drive (just to make sure Windows couldn't overwrite Linux). Got that working too.

Then I plugged the Linux drive in, as the first drive on the IDE cable; and the Windows disk as the second.

The trick then is to get grub (the Linux bootloader I'm using) to present you with both disks as options as boot time. There's a sample configuration in /boot/grub/menu.lst, but that didn't work for me: it looked like it was working, then just hung. I tried a couple of things, but nothing which worked.

Finally, I found this blog entry and used the configuration there. The trick is to make Windows think it's installed on the first disk on the IDE cable. I added this to the bottom of menu.lst:

title Windows 2000
rootnoverify (hd1,0)
map (hd0) (hd1)
map (hd1) (hd0)
chainloader +1

which does the trick! Now I get a working Windows 2000 option in my grub boot menu.

Creating a self-signed SSL certificate for Apache on Linux

Thu, 2009-03-05 10:36 — elliot

(This is extracted from my Apache course materials, but it's a useful howto in its own right.)

To generate a self-signed SSL certificate, you will need openssl installed first.

Then follow these steps:

  1. Generate the server's private key; we'll use a 1024-bit key using the RSA algorithm:
    openssl genrsa -out server.key 1024
  2. Generate a certificate-signing request:
    openssl req -new -key server.key -out server.csr
  3. Fill in the required information at the prompts:
       Country Name (2 letter code) [GB]:GB
   State or Province Name (full name) []:.
   Locality Name (eg, city) [Newbury]:Birmingham
   Organization Name (eg, company) [My Company Ltd]:Talis
   Organizational Unit Name (eg, section) []:Library Products
   Common Name (eg, your name or your server's hostname) []:prism.talis.com
   Email Address []:.

   Please enter the following 'extra' attributes to be sent with your certificate request

   A challenge password []:.
   An optional company name []:.
    The really important one is the Common Name: this must match the domain name which will serve the SSL site; otherwise connecting clients will get a prompt about a mismatch between the certificate's host name and the actual host name of the server.

    Note that we left the password blank. If we don't do this, Apache will prompt you for the certificate password each time you start the server, which is a pain in the arse.
  4. Create a self-signed certificate from the certificate-signing request (.csr file):
    openssl x509 -req -days 3650 -in server.csr -signkey server.key -out server.crt
  5. rm server.csr (you don't need it any more)
  6. Put the .crt and .key files into Apache's SSL directory and configure Apache to use them

If I get round to it I'll do another entry explaining how to make Apache use them.

Dealing with self-signed SSL certificates when running Selenium server with Firefox

Wed, 2009-01-14 18:58 — elliot

Selenium is a decent tool for testing web UIs, with good integration with a variety of languages. We use it on Talis Prism for testing the UI, running a Selenium server instance then firing Ruby rspec tests and an older HTML suite at it. Here's the part of the Ant build script which runs the HTML suite using Selenium :

<target name="prism-selenium-tests" description="Run the old Prism Selenium tests">
  <echo message="Running old Selenium tests against Prism" />
  <java jar="test/dependencies/Selenium/selenium-server.jar" fork="true" maxmemory="1024m">
    <arg line="-debug -timeout 500 -htmlSuite '*chrome ${firefox.bin}' http://${prism.host} \
       test/selenium/testSuite.html doc/seleniumResults.html" />
  </java>
</target>

where the variables we interpose are:

${firefox.bin} = path to the Firefox binary to use
${prism.host} = HTTP host to run the tests against

This works without a hitch if you're not using HTTPS; but as soon as your tests redirect to an HTTPS URL on the same host (we serve parts of Prism over SSL), where your SSL certificate is self-signed, things go wrong. As Selenium effectively runs Firefox with a new profile every time, you potentially lose any certificate exceptions you might accept.

One technique we were using was to create a custom profile; run Firefox using that profile; browse to the HTTPS URL and accept the exception into that profile; then close the profile.

This kind of worked, but we still got odd popups from Firefox about new extensions being installed. Just annoying.

I think I've now worked out the solution, which was largely based on http://kapanka.com/2008/12/selenium-rc-firefox-and-the-self-signed-ssl-c.... It's a bit of a pain in the arse, but it does seem to work. Here goes.

  1. Close down any running Firefox instances.
  2. Start Firefox (the one you're going to run your tests with) with the profile manager: firefox -ProfileManager
  3. Create a new profile. You'll be prompted to choose a directory for the profile. Put it somewhere inside the project where you're writing the tests.
  4. Select the profile and run Firefox using it.
  5. Browse to the HTTPS URL (with self-signed certificate) you're going to be testing against.
  6. Accept the self-signed certificate when prompted. This creates an exception for it in the profile.
  7. Close the browser.
  8. Go to the Firefox profile directory.
  9. Delete everything in the directory except for the cert_override.txt and cert8.db files.
  10. When you run your Selenium server (like in my Ant example above), pass a -firefoxProfileTemplate /path/to/profile/dir argument to it. This tells Selenium to use your partial profile (with certificate exceptions) as a basis for minting its new profile. So you get the certificate exceptions, but without any of the other clutter you would get if you used a whole profile.

The Ant task above, with this option, looks like this:

<target name="prism-selenium-tests" description="Run the old Prism Selenium tests">
  <echo message="Running old Selenium tests against Prism" />
  <java jar="test/dependencies/Selenium/selenium-server.jar" fork="true" maxmemory="1024m">
    <arg line="-debug -timeout 500 -firefoxProfileTemplate test/firefoxProfile \
       -htmlSuite '*chrome ${firefox.bin}' http://${prism.host} test/selenium/testSuite.html doc/seleniumResults.html" />
  </java>
</target>

Outside of Ant, the command might look something like:

java -jar test/dependencies/Selenium/selenium-server.jar -firefoxProfileTemplate /path/to/profile \
-htmlSuite '*chrome firefox-bin' http://host.com testSuite.html seleniumResults.html

Works for me.

Skype on Ubuntu Intrepid Ibex 8.10

Tue, 2009-01-06 13:27 — elliot

I upgraded my laptop just before Christmas, but unfortunately couldn't get Skype working easily.

First off, I tried the packages for Ubuntu provided by Skype, but I couldn't get them to work. Instead, I used the Medibuntu Skype package. To use these, you just add a couple of repositories to your apt sources: see this page for details.

These worked, kind of: Skype started, but the audio capture didn't work. I found this article, which suggests a fix, by turning off pulseaudio (the default, poorly-configured sound server for Intrepid).

But this didn't seem to completely solve my issues, as Skype kept hanging when you tried to close it down.

So, instead, I followed these longer but more comprehensive instructions, which leave pulseaudio turned on, but fix its configuration. They seem to have done the trick.

Installing Xdebug for PHP 5 on Ubuntu "Feisty Fawn"

Thu, 2008-06-19 08:56 — elliot

Xdebug is a PHP debugger with nice Eclipse integration. Here are some instructions for installing it (assuming you already have Apache 2 and PHP 5).

This article explains in more detail, but isn't Ubuntu-specific. It does detail Eclipse configuration for Xdebug in detail, though.

You need to be root to do the installation.

First off, install Xdebug. This isn't packaged for Ubuntu, so you need to do it with PECL. So install PECL if you don't have it:

apt-get install pecl

Use PECL to install Xdebug:

pecl install xdebug

Configure PHP 5 to use Xdebug by adding these lines to /etc/php5/apache2/php.ini (somewhere near where the other extension= lines are):

zend_extension=/usr/lib/php5/20060613+lfs/xdebug.so

[xdebug]
xdebug.remote_enable=1
xdebug.remote_handler=dbgp
xdebug.remote_mode=req
xdebug.remote_port=9000
xdebug.remote_host=127.0.0.1
xdebug.remote_log=/var/log/apache2/xdebug_remote.log

Note you need to use zend_extension= to load the extension, and you should use the absolute path to the module (.so file) to do this. Otherwise it fails.

Check using PHP info, e.g. add a file called phpinfo.php to your web root:

<?php phpinfo(); ?>

Then call it in your browser. Check that there is an Xdebug section displayed.

That's Xdebug installed. See the article linked at the start of this entry if you want to integrate with Eclipse.

12345next ›last »
Syndicate content