The folder I was trying to archive is about 72GB, but much like rsync at about 17GB it chokes because of the filesize. What's with so many common and essential Linux tools having such limitations? I guess it is likely that the authors never wrote their code with the idea that files would be so large but it's still very annoying. It's important to stay on top of these limitations on production servers because I didn't realize what happened until I checked the file with "gzip -t" and the archive was too short. I then ran the same script again on that directory and recorded the output as shown below from gzip.
time tar -czpf somefile.tar.gz archive-directory
gzip: stdout: File too large
Copyright 2002 Free Software Foundation
Copyright 1992-1993 Jean-loup Gailly
This program comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of this program
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING.
DIRENT UTIME STDC_HEADERS HAVE_UNISTD_H HAVE_MEMORY_H HAVE_STRING_H HAVE_LSTAT ASMV
Written by Jean-loup Gailly.
I don't know the specific limitations of other formats but I'm currently trying to tar and bzip2 the same directory and will report back. I wonder if rar is a viable option and what the maximum size is. At the very least, rar may make the most sense because you can easily split the archives into 4GB files and this also makes for easy DVD-R backups as well.
bzip2: I/O or other error, bailing out. Possible reason follows.
bzip2: File too large
Input file = (stdin), output file = (stdout)
tar, gzip, stdout, gbthe, folder, archive, gb, rsync, chokes, filesize, essential, linux, limitations, authors, servers, didn, quot, directory, recorded, output, czpf, somefile, gz, user, sys, copyright, software, loup, gailly, warranty, redistribute, copies, gnu, copying, compilation, dirent, utime, stdc_headers, have_unistd_h, have_memory_h, have_string_h, have_lstat, asmv, solutions, alternatives, formats, currently, bzip, rar, viable, maximum, archives, dvd, backups, slower, compression, bailing, follows, input, stdin,