wget -e robots=off It is as simple as the above and this is something one must watch out carefully when using wget because you may think you have archived or downloaded content when you never did due to a nofollow/robots.txt statement.........

There are a few ways of doing this and all basically involve using the reverse proxy or "ProxyPass" feature of Apache to accomplish it. 1.) Create a normal vhost and simply symlink the root directory of the site you want to mirror. Eg. originalsite.com and newsite.com /vhosts/originalsite.com/httpdocs You would symlink like this: ln -s /vhosts/originalsite.com/httpdocs vhosts/originalsite.com/........