Login and download all files script

This is very handy if you're too busy and don't have time to download whatever files you need.

The -D specifies the domains allowed, this is because I specified -H which means foreign hosts are allowed, if you don't restrict them you'll end up going to the whole internet via ads and other links just like a search Engine would follow.

-l 0 specifies to go deep, to as many levels as possible/as exist.

-e robots=off is important because robots.txt often says you can't view certain files which we don't want (this is intended for search engines).

-A .zip specifies that we only want to download .zip files, you can add more, change it from zip to jpg or remove it all together if you like.

--user and --password are hopefully obvious

wget -l 0 -e robots=off --keep-session-cookies -A .zip -H -D allowedomain.com,otherdomain.com -U "Mozilla/5.0 (X11; Linux i686; rv:12.0) Gecko/20100101 Firefox/12.0" -m --user=user --password=pass http://url.com

 


Tags:

login, download, scriptthis, specifies, domains, specified, hosts, restrict, ll, via, ads, links, levels, robots, txt, engines, zip, jpg, user, password, wget, allowedomain, otherdomain, quot, mozilla, linux, rv, gecko, firefox, http, url,

Latest Articles

  • Linux Ubuntu Cannot Print Large Images
  • Cannot Print PDF Solution and Howto Resize
  • Linux Console Login Screen TTY Change Message
  • Apache Cannot Start Listening Already on 0.0.0.0
  • MySQL Bash Query to pipe input directly without using heredoc trick
  • CentOS 6 and 7 / RHEL Persistent DHCP Solution
  • Debian Ubuntu Mint rc-local service startup error solution rc-local.service: Failed at step EXEC spawning /etc/rc.local: Exec format error
  • MySQL Cheatsheet Guide and Tutorial
  • bash script kill whois or other command that is running for too long
  • Linux tftp listens on all interfaces and IPs by DEFAULT Security Risk Hole Solution
  • python import docx error
  • Cisco Unified Communications Manager Express Cheatsheet CUCME CME
  • Linux Ubuntu Debian Missing privilege separation directory: /var/run/sshd
  • bash how to count the number of columns or words in a line
  • bash if statement how to test program output without assigning to variable
  • RTNETLINK answers: Network is unreachable
  • Centos 7 how to save iptables rules like Centos 6
  • nfs tuning maximum amount of connections
  • qemu-kvm error "Could not initialize SDL(No available video device) - exiting"
  • Centos 7 tftpd will not work with selinux enabled