wget and downloading the entire directory
Sometimes I need to retrieve the whole remote directory. Normally, using graphical user interface is the simplest way to download the directory, however, sometimes we do not prefer to use or we have no chance to use graphical user interfaces like working with GNU Linux servers without any desktop environments. In that case, we can use wget to get the whole directory. Most of you heard something about wget and now I will give you some specific information about it which helps you to download a directory from remote URL. I should thank you “Andrea Ben Benini” for this useful tip.
wget -r –level=0 -E –ignore-length -x -k -p -erobots=off -np -N http://www.shibumidojo.org/something/directory
Here are the options:
-r : Recursive retrieving (important)
–level=0: Specify recursion maximum depth level (0 for no limit), very important
-E: append “.html” extension to every document declared as “application/html”
useful when you deal with dirs (that are not dirs but index.html files)
–ignore-lenght: Ignore “Content-length” http headers, sometimes useful when dealing with bugged CGI programs
-x: Force dirs, create an hierarchy of directories even if one would not been created otherwise
-k: here’s one of the most useful options, it converts remote links to local for best viewing
-p: download ll the files that are necessary for proper display of the page
(not so reliable when dealing with JS code but useful)
-erobots=off: turn off http robots.txt usage
-np: no parent, do not ascend to parent dir when retrieving recursively,
one of the most useful function I’ve seen
Eğer yazıyı beğendiyseniz ya da ekleyecekleriniz varsa, lütfen yorumunuz yazın veya RSS aboneliği ile yeni yazılardan anında haberdar olun.
Yorumlar
Henüz Yorum Yok.
Yorum Yazın