wget and downloading the entire directory

CorpusCallosum — Sun, 28 Jun 2009 14:44:02 +0000

Sometimes I need to retrieve the whole remote directory. Normally, using graphical user interface is the simplest way to download the directory, however, sometimes we do not prefer to use or we have no chance to use graphical user interfaces like working with GNU Linux servers without any desktop environments. In that case, we can use wget to get the whole directory. Most of you heard something about wget and now I will give you some specific information about it which helps you to download a directory from remote URL. I should thank you “Andrea Ben Benini” for this useful tip.

wget -r –level=0 -E –ignore-length -x -k -p -erobots=off -np -N http://www.shibumidojo.org/something/directory

Here are the options:

-r : Recursive retrieving (important)
–level=0: Specify recursion maximum depth level (0 for no limit), very important
-E: append “.html” extension to every document declared as “application/html”
useful when you deal with dirs (that are not dirs but index.html files)
–ignore-lenght: Ignore “Content-length” http headers, sometimes useful when dealing with bugged CGI programs
-x: Force dirs, create an hierarchy of directories even if one would not been created otherwise
-k: here’s one of the most useful options, it converts remote links to local for best viewing
-p: download ll the files that are necessary for proper display of the page
(not so reliable when dealing with JS code but useful)
-erobots=off: turn off http robots.txt usage
-np: no parent, do not ascend to parent dir when retrieving recursively,
one of the most useful function I’ve seen

Shibumi Dojo » wget

wget and downloading the entire directory