tar gzip: stdout: File too large - Error Size Limit on Large file > 17GB

The folder I was trying to archive is about 72GB, but much like rsync at about 17GB it chokes because of the filesize.  What's with so many common and essential Linux tools having such limitations?  I guess it is likely that the authors never wrote their code with the idea that files would be so large but it's still very annoying.  It's important to stay on top of these limitations on production servers because I didn't realize what happened until I checked the file with "gzip -t" and the archive was too short.  I then ran the same script again on that directory and recorded the output as shown below from gzip.

time tar -czpf somefile.tar.gz archive-directory

gzip: stdout: File too large

real    33m15.478s
user    0m2.440s
sys     0m34.498s

 

gzip -V
gzip 1.3.5
(2002-09-30)
Copyright 2002 Free Software Foundation
Copyright 1992-1993 Jean-loup Gailly
This program comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of this program
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING.
Compilation options:
DIRENT UTIME STDC_HEADERS HAVE_UNISTD_H HAVE_MEMORY_H HAVE_STRING_H HAVE_LSTAT ASMV
Written by Jean-loup Gailly.

 

Solutions/Alternatives to gzip?

I don't know the specific limitations of other formats but I'm currently trying to tar and bzip2 the same directory and will report back.  I wonder if rar is a viable option and what the maximum size is.  At the very least, rar may make the most sense because you can easily split the archives into 4GB files and this also makes for easy DVD-R backups as well.

bzip2 had the same error at the 17GB mark (and this also shows how much SLOWER bzip2 is with hardly much better compression):

bzip2: I/O or other error, bailing out.  Possible reason follows.
bzip2: File too large
        Input file = (stdin), output file = (stdout)

real    199m7.113s
user    0m2.200s
sys     0m29.714s


Tags:

tar, gzip, stdout, gbthe, folder, archive, gb, rsync, chokes, filesize, essential, linux, limitations, authors, servers, didn, quot, directory, recorded, output, czpf, somefile, gz, user, sys, copyright, software, loup, gailly, warranty, redistribute, copies, gnu, copying, compilation, dirent, utime, stdc_headers, have_unistd_h, have_memory_h, have_string_h, have_lstat, asmv, solutions, alternatives, formats, currently, bzip, rar, viable, maximum, archives, dvd, backups, slower, compression, bailing, follows, input, stdin,

Latest Articles

  • How To Upgrade Debian 8,9,10 to Debian 12 Bookworm
  • Linux dhcp dhclient Mint Redhat Ubuntu Debian How To Use Local Domain DNS Server Instead of ISPs
  • Docker dockerd swarm high CPU usage cause solution
  • Docker Minimum Requirements/How Efficient is Docker? How Much Memory Does Dockerd Use?
  • qemu-nbd: Failed to set NBD socket solution qemu-nbd: Disconnect client, due to: Failed to read request: Unexpected end-of-file before all bytes were read
  • apache2 httpd apache server will not start [pid 22449:tid 139972160445760] AH00052: child pid 23248 exit signal Aborted (6) solution Mint Debian Ubuntu Redhat
  • How to use the FTDI USB serial cable to RJ45 adapter to connect to the console on Cisco/Juniper Switch Router Firewall in Linux Ubuntu Debian Redhat
  • How To Setup Python3 in Ubuntu Docker Image for AI Deep Learning
  • How to Configure NVIDIA GPUs with Docker on Ubuntu: A Comprehensive Guide for AI Deep Learning CUDA Solution
  • Linux Ubuntu Mint how to check nameservers when /etc/resolv.conf disabled solution
  • Docker cannot work on other overlayfs filesystems such as ecryptfs won't start overlayfs: filesystem on '/home/docker/overlay2/check-overlayfs-support130645871/upper' not supported as upperdir
  • Linux How To Access Original Contents of Directory Mounted Debian Mint CentOS Redhat Solution
  • ecryptfs how to manually encrypt your existing home directory or other directory
  • How to Reset CIPC Cisco IP Communicator for CME CUCM CallManager
  • Internet Explorer Cannot Download File "Your security settings do not allow for this file to be downloaded." Security Settings Solution
  • Linux How To Upgrade To The Latest Kernel Debian Mint Ubuntu
  • Firefox how to restore and backup saved passwords and history which files/location
  • Linux How To echo as root solution to use tee permission denied solution Ubuntu Debian Mint Redhat CentOS
  • Linux how to keep command line bash process running if you are disconnected or need to logout of SSH remotely
  • Linux swapping too much? How to check the swappiness and stop swapping