wget ignore robots.txt howto
wget -e robots=off
It is as simple as the above and this is something one must watch out carefully when using wget because you may think you have archived or downloaded content when you never did due to a nofollow/robots.txt statement.
Tags:
wget, robots, txt, howto, archived, downloaded, content, nofollow,