Occasionally you need to mirror a website (or a directory inside one). If you've only got HTTP access, there are tools like httrack which are pretty good (albeit pretty ugly) at doing this. However, as far as I can tell, you can't use httrack on a password-protected website.
curl can probably do this too, and supports authentication, but it wasn't obvious.
So I ended up using wget, as it supports mirroring and credentials. But the issue here is that wget plays nice and respects robots.txt; which can actually prevent you mirroring a site you own. And nothing in the man page explains how to ignore robots.txt.
Eventually, I came up with this incantation, which works for me (access to password-protected site, full mirror, ignoring robots.txt):
wget -e robots=off --wait 1 -x --user=xxx --password=xxx -m -k http://domain.to.mirror/
where:
Don't use it carelessly on someone else's website, as they might get angry...
I'm not going to try to explain RDF and/or RDFa here, but thought any poor suckers looking for RDFa examples might benefit from me posting what I finally worked out, with help from my colleague Rob. Namely, how to annotate an HTML ordered list (<ol>) with RDFa attributes; and how to put RDFa attributes onto form elements.
Here's the HTML page with RDFa embedded in it. What I'm representing here is a sequence of collections, and the individual collections within it:
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML+RDFa 1.0//EN" "http://www.w3.org/MarkUp/DTD/xhtml-rdfa-1.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <title>Collections</title> </head> <body> <h1>Collections</h1> <form method="post" action="http://receptacular.org/collections"> <ol xmlns="http://www.w3.org/1999/xhtml" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:xsd="http://www.w3.org/2001/XMLSchema#" xmlns:rec="http://receptacular.org/schema#" typeof="rdf:Seq" about="http://receptacular.org/collections"> <li rel="rdf:_1" resource="http://receptacular.org/collections/1"> <span style="display:none;" rel="rdf:type" resource="http://receptacular.org/schema#Collection"></span> <div class="collection-label" property="rdfs:label">Vague Collection</div> <input type="checkbox" id="collections-1-hidden" property="rec:hidden" datatype="xsd:boolean" content="false"/> <label for="collections-1-hidden">hidden</label> <input type="checkbox" id="collections-1-defaultSearch" property="rec:defaultSearch" datatype="xsd:boolean" content="false"/> <label for="collections-1-defaultSearch">use for searches</label> </li> <li rel="rdf:_2" resource="http://receptacular.org/collections/2"> <span style="display:none;" rel="rdf:type" resource="http://receptacular.org/schema#Collection"></span> <div class="collection-label" property="rdfs:label">Archive Collection</div> <input type="checkbox" id="collections-2-hidden" property="rec:hidden" datatype="xsd:boolean" content="false"/> <label for="collections-2-hidden">hidden</label> <input type="checkbox" id="collections-2-defaultSearch" property="rec:defaultSearch" datatype="xsd:boolean" content="false"/> <label for="collections-2-defaultSearch">use for searches</label> </li> <li rel="rdf:_3" resource="http://receptacular.org/collections/3"> <span style="display:none;" rel="rdf:type" resource="http://receptacular.org/schema#Collection"></span> <div class="collection-label" property="rdfs:label">Main Collection</div> <input type="checkbox" id="collections-3-hidden" property="rec:hidden" datatype="xsd:boolean" content="true" checked="checked"/> <label for="collections-3-hidden">hidden</label> <input type="checkbox" id="collections-3-defaultSearch" property="rec:defaultSearch" datatype="xsd:boolean" content="true" checked="checked"/> <label for="collections-3-defaultSearch">use for searches</label> </li> </ol> <p> <input type="button" value="Save" id="save-collections"/> </p> </form> </body> </html>
Available online here: http://receptacular.org/collections
Things of note:
To see the RDF which can be extracted from this page, you can use the W3C's RDFa Distiller. Here's the resulting RDF:
<?xml version="1.0" encoding="utf-8"?> <rdf:RDF xmlns:dist="http://www.w3.org/2007/08/pyRdfa/distiller#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#" xmlns:rec="http://receptacular.org/schema#" xmlns:xhv="http://www.w3.org/1999/xhtml/vocab#" xmlns:xml="http://www.w3.org/XML/1998/namespace" xmlns:xsd="http://www.w3.org/2001/XMLSchema#" > <rdf:Seq rdf:about="http://receptacular.org/collections"> <rdf:_1> <rec:Collection rdf:about="http://receptacular.org/collections/1"> <rec:hidden rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">false</rec:hidden> <rec:defaultSearch rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">false</rec:defaultSearch> <rdfs:label>Vague Collection</rdfs:label> </rec:Collection> </rdf:_1> <rdf:_2> <rec:Collection rdf:about="http://receptacular.org/collections/2"> <rec:hidden rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">false</rec:hidden> <rec:defaultSearch rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">false</rec:defaultSearch> <rdfs:label>Archive Collection</rdfs:label> </rec:Collection> </rdf:_2> <rdf:_3> <rec:Collection rdf:about="http://receptacular.org/collections/3"> <rec:hidden rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">true</rec:hidden> <rec:defaultSearch rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">true</rec:defaultSearch> <rdfs:label>Main Collection</rdfs:label> </rec:Collection> </rdf:_3> </rdf:Seq> </rdf:RDF>
Saving changes to an RDFa-enabled form like this is another challenge, for which we used rdfquery, and RDF library for JQuery. (I recommend you use the latest svn HEAD version of this library, as older versions have a bug where they ignore RDFa elements nested inside elements without RDFa attributes.) Maybe I'll get round to that another time.
This is a Ruby script which randomly copies mp3 files from one directory to an mp3 player. I wrote it so I could easily fill up my mp3 player from the 9000 odd mp3s I have on a different external drive.
To run it, you'll need the sys-filesystem gem (see http://rubyforge.org/projects/sysutils):
$ gem install sys-filesystem
Next, edit these variables in the script (near the top):
* source_dir
to the directory containing the mp3s you want to select from
* dest_dir
to the path for the directory on your mp3 player you want to copy to
Be a bit careful, as this will attempt to fill the dest_dir
you specify with mp3 files from source_dir
. You might end up filling the wrong disk up.
Then just run it with ruby from the command line:
$ ruby mp3s_random.rb
Note that it won't delete anything from the destination drive, and will attempt to fill all the space available. Also note that it doesn't keep trying mp3s until it finds one which will fit the last remaining space: once it tries to copy a file which won't fit, it stops. You can always run it again to see whether the next run finds a file small enough to fit.
I've only tested it on Linux, but, who knows, it might work on Windows too. (No operating-system specific commands are used in the script, as it uses Ruby for all file operations.)
The code is below, but I've attached it as well.
require 'rubygems' require 'sys/filesystem' require 'ftools' source_dir = '/media/disk/music' dest_dir = '/media/disk-1/music' files = Dir[File.join(source_dir, '/**/*.mp3')] num_files = files.size stat = Sys::Filesystem.stat(dest_dir) disk_free_space_kb = (stat.blocks_free * stat.fragment_size).to_kb files_selected = [] while disk_free_space_kb > 0 and num_files > 0 do # choose an mp3 file_num_to_copy = rand(num_files - 1) + 1 file_path = files.delete_at(file_num_to_copy) num_files = num_files - 1 # work out how big file is file_size_kb = File.stat(file_path).size.to_kb # subtract from free space if (disk_free_space_kb - file_size_kb) > 0 files_selected << file_path disk_free_space_kb = disk_free_space_kb - file_size_kb else break end end files_selected.each do |f| copy_to_path = File.join(dest_dir, File.basename(f)) puts "Copying #{f} to device" File.copy(f, copy_to_path) end
I'm using Ubuntu Intrepid Ibex, but the ffmpeg distribution it comes with doesn't support recent FLV file encodings (like some videos from YouTube). You get an error like this when you try to do anything with them:
[flv @ 0xb800e4c8]Unsupported video codec (7)
My solution was to checkout ffmpeg from its Subversion repository and compile it myself:
$ svn checkout svn://svn.ffmpeg.org/ffmpeg/trunk ffmpeg-svn $ cd ffmpeg-svn $ ./configure --prefix=~/apps/ffmpeg-svn --enable-libmp3lame $ make install
To do the compile, you'll probably need build-essential, as well as libmp3lame-dev, and the *-dev versions of any other codecs you want to use.
Use the resulting binary to do the conversion:
$ ~/apps/ffmpeg-svn/bin/ffmpeg -i infile.flv outfile.mpg
It turns out this version does support those recent FLV files, happily.
If you want to do something like a SQL NOT in SPARQL, here's what the query looks like:
PREFIX rs: <http://schemas.talis.com/2006/recordstore/schema#> PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> DESCRIBE ?tenancy { ?tenancy rdf:type rs:Tenancy . OPTIONAL { ?tenancy rs:platformStoreUri ?o } . FILTER ( !bound(?o) ) }
Here I'm looking for subjects with rdf type of http://schemas.talis.com/2006/recordstore/schema#Tenancy, which don't have a http://schemas.talis.com/2006/recordstore/schema#platformStoreUri predicate. The important bit is that you make the predicate which could potentially not be "set" OPTIONAL; and add a FILTER which only includes subjects where the predicate is bound to a value. This effectively screens out any subjects where the predicate has not been added to the subject. This pattern is basically Negation as Failure (according to the SPARQL recommendation), which derives from logic programming. Feels a bit like being back at university.
I recently upgraded the hardware of my old desktop PC, with the aim of providing the house with a new-ish Linux machine for watching movies and using the internet, and a Windows machine for writing music and playing (old) games. My plan was to use two hard disks: one for Linux, another for Windows, and choose which to use at boot time.
Normal procedure is to install Windows first, then install Linux into a spare partition on the same hard drive (Windows tends to overwrite any disk you put it on). But it's easier to get a Linux machine up and running, see what hardware you've got, and get a decent system without needing to go and find loads of old drivers. So I decided to install Linux first. I plugged in a drive for it as the Primary IDE drive, and installed Ubuntu Linux onto it.
Then, I unplugged the Linux drive, plugged the other drive in, and installed Windows 2000 onto the second drive (just to make sure Windows couldn't overwrite Linux). Got that working too.
Then I plugged the Linux drive in, as the first drive on the IDE cable; and the Windows disk as the second.
The trick then is to get grub (the Linux bootloader I'm using) to present you with both disks as options as boot time. There's a sample configuration in /boot/grub/menu.lst, but that didn't work for me: it looked like it was working, then just hung. I tried a couple of things, but nothing which worked.
Finally, I found this blog entry and used the configuration there. The trick is to make Windows think it's installed on the first disk on the IDE cable. I added this to the bottom of menu.lst:
title Windows 2000 rootnoverify (hd1,0) map (hd0) (hd1) map (hd1) (hd0) chainloader +1
which does the trick! Now I get a working Windows 2000 option in my grub boot menu.
(This is extracted from my Apache course materials, but it's a useful howto in its own right.)
To generate a self-signed SSL certificate, you will need openssl installed first.
Then follow these steps:
openssl genrsa -out server.key 1024
openssl req -new -key server.key -out server.csr
Country Name (2 letter code) [GB]:GB State or Province Name (full name) []:. Locality Name (eg, city) [Newbury]:Birmingham Organization Name (eg, company) [My Company Ltd]:Talis Organizational Unit Name (eg, section) []:Library Products Common Name (eg, your name or your server's hostname) []:prism.talis.com Email Address []:. Please enter the following 'extra' attributes to be sent with your certificate request A challenge password []:. An optional company name []:.The really important one is the Common Name: this must match the domain name which will serve the SSL site; otherwise connecting clients will get a prompt about a mismatch between the certificate's host name and the actual host name of the server.
openssl x509 -req -days 3650 -in server.csr -signkey server.key -out server.crt
rm server.csr
(you don't need it any more)If I get round to it I'll do another entry explaining how to make Apache use them.
Selenium is a decent tool for testing web UIs, with good integration with a variety of languages. We use it on Talis Prism for testing the UI, running a Selenium server instance then firing Ruby rspec tests and an older HTML suite at it. Here's the part of the Ant build script which runs the HTML suite using Selenium :
<target name="prism-selenium-tests" description="Run the old Prism Selenium tests"> <echo message="Running old Selenium tests against Prism" /> <java jar="test/dependencies/Selenium/selenium-server.jar" fork="true" maxmemory="1024m"> <arg line="-debug -timeout 500 -htmlSuite '*chrome ${firefox.bin}' http://${prism.host} \ test/selenium/testSuite.html doc/seleniumResults.html" /> </java> </target>
where the variables we interpose are:
${firefox.bin} = path to the Firefox binary to use ${prism.host} = HTTP host to run the tests against
This works without a hitch if you're not using HTTPS; but as soon as your tests redirect to an HTTPS URL on the same host (we serve parts of Prism over SSL), where your SSL certificate is self-signed, things go wrong. As Selenium effectively runs Firefox with a new profile every time, you potentially lose any certificate exceptions you might accept.
One technique we were using was to create a custom profile; run Firefox using that profile; browse to the HTTPS URL and accept the exception into that profile; then close the profile.
This kind of worked, but we still got odd popups from Firefox about new extensions being installed. Just annoying.
I think I've now worked out the solution, which was largely based on http://kapanka.com/2008/12/selenium-rc-firefox-and-the-self-signed-ssl-c.... It's a bit of a pain in the arse, but it does seem to work. Here goes.
firefox -ProfileManager
-firefoxProfileTemplate /path/to/profile/dir
argument to it. This tells Selenium to use your partial profile (with certificate exceptions) as a basis for minting its new profile. So you get the certificate exceptions, but without any of the other clutter you would get if you used a whole profile.The Ant task above, with this option, looks like this:
<target name="prism-selenium-tests" description="Run the old Prism Selenium tests"> <echo message="Running old Selenium tests against Prism" /> <java jar="test/dependencies/Selenium/selenium-server.jar" fork="true" maxmemory="1024m"> <arg line="-debug -timeout 500 -firefoxProfileTemplate test/firefoxProfile \ -htmlSuite '*chrome ${firefox.bin}' http://${prism.host} test/selenium/testSuite.html doc/seleniumResults.html" /> </java> </target>
Outside of Ant, the command might look something like:
java -jar test/dependencies/Selenium/selenium-server.jar -firefoxProfileTemplate /path/to/profile \ -htmlSuite '*chrome firefox-bin' http://host.com testSuite.html seleniumResults.html
Works for me.
I upgraded my laptop just before Christmas, but unfortunately couldn't get Skype working easily.
First off, I tried the packages for Ubuntu provided by Skype, but I couldn't get them to work. Instead, I used the Medibuntu Skype package. To use these, you just add a couple of repositories to your apt sources: see this page for details.
These worked, kind of: Skype started, but the audio capture didn't work. I found this article, which suggests a fix, by turning off pulseaudio (the default, poorly-configured sound server for Intrepid).
But this didn't seem to completely solve my issues, as Skype kept hanging when you tried to close it down.
So, instead, I followed these longer but more comprehensive instructions, which leave pulseaudio turned on, but fix its configuration. They seem to have done the trick.
Xdebug is a PHP debugger with nice Eclipse integration. Here are some instructions for installing it (assuming you already have Apache 2 and PHP 5).
This article explains in more detail, but isn't Ubuntu-specific. It does detail Eclipse configuration for Xdebug in detail, though.
You need to be root to do the installation.
First off, install Xdebug. This isn't packaged for Ubuntu, so you need to do it with PECL. So install PECL if you don't have it:
apt-get install pecl
Use PECL to install Xdebug:
pecl install xdebug
Configure PHP 5 to use Xdebug by adding these lines to /etc/php5/apache2/php.ini (somewhere near where the other extension=
lines are):
zend_extension=/usr/lib/php5/20060613+lfs/xdebug.so [xdebug] xdebug.remote_enable=1 xdebug.remote_handler=dbgp xdebug.remote_mode=req xdebug.remote_port=9000 xdebug.remote_host=127.0.0.1 xdebug.remote_log=/var/log/apache2/xdebug_remote.log
Note you need to use zend_extension=
to load the extension, and you should use the absolute path to the module (.so file) to do this. Otherwise it fails.
Check using PHP info, e.g. add a file called phpinfo.php to your web root:
<?php phpinfo(); ?>
Then call it in your browser. Check that there is an Xdebug section displayed.
That's Xdebug installed. See the article linked at the start of this entry if you want to integrate with Eclipse.