Most of the Administrators are fire fighting to take the load off database servers which directly makes an impact on Web application server. In case of enormous website with lot of old data or data in archive, it is better to convert it to static rather then serving from Database, A single crawler may crawl lot of old data, not effective for traffic but still will consume lot of server resources.
Also sometimes users need to download website to read the data later when not connected to internet
Lets analyze some of the most popular software which can be used to take the static dump of a website :
1. Wget : Wget is the most popular command line utility used to take the dump of website, usually in linux and unix based operating systems ,Windows based version is also available.It has extensive range of option which can perform complex operation and is also used as a crawler.
2. Curl : Curl is also a famous utility which is more complex and can perform more operations as compared to wget ,usually in most cases wget is enough but sometimes to achieve custom requirements curl can rank best then any other tool, Many crawlers, web transfer and cache based API and codes are written using curl APIs.
3. HTTRACK : HTTRACK is a famous website copier utility which is available in both GUI and CLI and available for multiple operating system inclusing windows, linux, BSD and Unix . It is easier to install and configure and is a good toold for beginners.
4. GetLeft: Getleft is a very small utility from website grabbing and making website static ,It is only available on windows as a GUI client.
5. SiteSucker : We cannot leave Mac users out , Can we ? Sitesucker is a Mac based App which can download whole website and files.Great thing is it can be installed on iPhone and iPad also.
