townx - Using wget to mirror a website - Comments

Hello

san antonio dentists — Sat, 24 Dec 2011 10:52:07 -0600

Great stuff of your stuff, man. I’ve read your stuff before and you’re too awesome. I enjoy what you’ve got here, love what you’re saying and exactly how you say it. You create it entertaining but you just seem to ensure that is stays smart. I can’t wait to see more of your stuff. This is really a fantastic blog. san antonio dentists

Well,

PaxSkeptica — Fri, 25 Nov 2011 22:10:08 -0600

in that case, +1 because I was also trying to mirror a site [admittedly one that I do not own, but also part of a rather large "cloud" site so I don't feel too bad mirroring a small 80-page site ignoring robots.txt, or at least so I tell myself] & couldn't seem to get out of wget "playing nice" mode.

I put a short (and probably

elliot — Sat, 29 May 2010 06:49:24 -0500

I put a short (and probably not very helpful) suggestion in reply to your other comment. Always happy to get a genuine comment from someone who found my blog useful!

Nifty use of wget

John — Wed, 19 May 2010 09:26:51 -0500

Nifty use of wget, Seems so simple and useful I, not only bookmarked it but also cut & pasted your article in my personal linux help document.. Dont want to risk a page not found some time off in the distant future. ;)
Consequentially I stumbled onto your blog searching for an easy way to implement spam filtering. The how to was very helpful, but left me with one question when training my inbox (see my comment in that blog entry, if you have any insight)
Thanks, John

Using wget to mirror a website

elliot — Wed, 12 May 2010 04:29:46 -0500

Occasionally you need to mirror a website (or a directory inside one). If you've only got HTTP access, there are tools like httrack which are pretty good (albeit pretty ugly) at doing this. However, as far as I can tell, you can't use httrack on a password-protected website.

curl can probably do this too, and supports authentication, but it wasn't obvious.

So I ended up using wget, as it supports mirroring and credentials. But the issue here is that wget plays nice and respects robots.txt; which can actually prevent you mirroring a site you own. And nothing in the man page explains how to ignore robots.txt.

Eventually, I came up with this incantation, which works for me (access to password-protected site, full mirror, ignoring robots.txt):

wget -e robots=off --wait 1 -x --user=xxx --password=xxx -m -k http://domain.to.mirror/

where:

-e robots=off obviously disables robots
--wait 1 forces a pause between gets (so the site doesn't get hammered)
--user and --password: self-evident
-x creates a local directory structure which "mirrors" (see what I did there?) the directory structure on the site you're mirroring
-m turns on mirror mode: "turns on recursion and time-stamping, sets infinite recursion depth and keeps FTP directory listings" (from the man page)
-k converts links after download so that URLs in the mirrored files reference local files

Don't use it carelessly on someone else's website, as they might get angry...