Using wget to mirror a website

Occasionally you need to mirror a website (or a directory inside one). If you've only got HTTP access, there are tools like httrack which are pretty good (albeit pretty ugly) at doing this. However, as far as I can tell, you can't use httrack on a password-protected website.

curl can probably do this too, and supports authentication, but it wasn't obvious.

So I ended up using wget, as it supports mirroring and credentials. But the issue here is that wget plays nice and respects robots.txt; which can actually prevent you mirroring a site you own. And nothing in the man page explains how to ignore robots.txt.

Eventually, I came up with this incantation, which works for me (access to password-protected site, full mirror, ignoring robots.txt):

wget -e robots=off --wait 1 -x --user=xxx --password=xxx -m -k http://domain.to.mirror/

where:

  • -e robots=off obviously disables robots
  • --wait 1 forces a pause between gets (so the site doesn't get hammered)
  • --user and --password: self-evident
  • -x creates a local directory structure which "mirrors" (see what I did there?) the directory structure on the site you're mirroring
  • -m turns on mirror mode: "turns on recursion and time-stamping, sets infinite recursion depth and keeps FTP directory listings" (from the man page)
  • -k converts links after download so that URLs in the mirrored files reference local files

Don't use it carelessly on someone else's website, as they might get angry...

Comments

Nifty use of wget

Nifty use of wget, Seems so simple and useful I, not only bookmarked it but also cut & pasted your article in my personal linux help document.. Dont want to risk a page not found some time off in the distant future. ;)
Consequentially I stumbled onto your blog searching for an easy way to implement spam filtering. The how to was very helpful, but left me with one question when training my inbox (see my comment in that blog entry, if you have any insight)
Thanks, John

I put a short (and probably

I put a short (and probably not very helpful) suggestion in reply to your other comment. Always happy to get a genuine comment from someone who found my blog useful!