From the wget manual
wget -r -nd --delete-after --no-parent --no-cookies http://whatever.com/~popular/page/
where>
-r -- retrieve recursively
-nd -- don't create directories
--delete-after -- delete from local machine after download
--relative -- follow elative links only
--no-parent -- only the files below a certain hierarchy will be downloaded
--save-headers -- save the server headers with the file
--wait=5 -- wait 5 seconds between each fetch
-nd -- don't create directories
--delete-after -- delete from local machine after download
--relative -- follow elative links only
--no-parent -- only the files below a certain hierarchy will be downloaded
--save-headers -- save the server headers with the file
--wait=5 -- wait 5 seconds between each fetch
Then have a look at your web server access log and check for 404 error messages.
