tar gzip: stdout: File too large - Error Size Limit on Large file > 17GB

The folder I was trying to archive is about 72GB, but much like rsync at about 17GB it chokes because of the filesize.  What's with so many common and essential Linux tools having such limitations?  I guess it is likely that the authors never wrote their code with the idea that files would be so large but it's still very annoying.  It's important to stay on top of these limitations on production servers because I didn't realize what happened until I checked the file with "gzip -t" and the archive was too short.  I then ran the same script again on that directory and recorded the output as shown below from gzip.

time tar -czpf somefile.tar.gz archive-directory

gzip: stdout: File too large

real    33m15.478s
user    0m2.440s
sys     0m34.498s

 

gzip -V
gzip 1.3.5
(2002-09-30)
Copyright 2002 Free Software Foundation
Copyright 1992-1993 Jean-loup Gailly
This program comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of this program
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING.
Compilation options:
DIRENT UTIME STDC_HEADERS HAVE_UNISTD_H HAVE_MEMORY_H HAVE_STRING_H HAVE_LSTAT ASMV
Written by Jean-loup Gailly.

 

Solutions/Alternatives to gzip?

I don't know the specific limitations of other formats but I'm currently trying to tar and bzip2 the same directory and will report back.  I wonder if rar is a viable option and what the maximum size is.  At the very least, rar may make the most sense because you can easily split the archives into 4GB files and this also makes for easy DVD-R backups as well.

bzip2 had the same error at the 17GB mark (and this also shows how much SLOWER bzip2 is with hardly much better compression):

bzip2: I/O or other error, bailing out.  Possible reason follows.
bzip2: File too large
        Input file = (stdin), output file = (stdout)

real    199m7.113s
user    0m2.200s
sys     0m29.714s


Tags:

tar, gzip, stdout, gbthe, folder, archive, gb, rsync, chokes, filesize, essential, linux, limitations, authors, servers, didn, quot, directory, recorded, output, czpf, somefile, gz, user, sys, copyright, software, loup, gailly, warranty, redistribute, copies, gnu, copying, compilation, dirent, utime, stdc_headers, have_unistd_h, have_memory_h, have_string_h, have_lstat, asmv, solutions, alternatives, formats, currently, bzip, rar, viable, maximum, archives, dvd, backups, slower, compression, bailing, follows, input, stdin,

Latest Articles

  • How to install Windows or other OS and then bring to another computer by using a physical drive and Virtual Machine with QEMU
  • PXE-E23 Error BOOTx64.EFI GRUB booting is 0 bytes tftp pxe dhcp solution NBP filesize is 0 Bytes
  • vagrant install on Debian Mint Ubuntu Linux RHEL Quick Setup Guide Tutorial
  • RHEL 8 CentOS 8, Alma Linux 8, Rocky Linux 8 System Not Booting with RAID or on other servers/computers Solution for dracut and initramfs missing kernel modules
  • How to Upgrade to Debian 11 from Version 8,9,10
  • Ubuntu Linux Mint Debian Redhat Cannot View Files on Android iPhone USB File Transfer Not Working Solution
  • Virtualbox Best Networking Mode In Lab/Work Environment without using NAT Network or Bridged
  • debootstrap how to install Ubuntu, Mint, Debian install
  • Linux grub not using UUID for the root device instead it uses /dev/sda1 or other device name solution
  • How To Restore Partition Table on Running Linux Mint Ubuntu Debian Machine
  • Debian Ubuntu apt install stop daemon questions/accept the default action without prompting
  • iptables NAT how to enable PPTP in newer Debian/Ubuntu/Mint Kernels Linux
  • Grandstream Phone Vulnerability Security Issue Remote Backdoor Connection to 207.246.119.209:3478
  • Linux How to Check Which NIC is Onboard eth0 or eth1 Ubuntu Centos Debian Mint
  • VboxManage VirtualBox NAT Network Issues Managment Troubleshooting
  • Dell PowerEdge Server iDRAC Remote KVM/IP Default Username, Password Reset and Login Information Solution
  • Nvidia Tesla GPUs K40/K80/M40/P40/P100/V100 at home/desktop hacking, cooling, powering, cable solutions Tutorial AIO Solutions
  • Stop ls in Linux Debian Mint CentOS Ubuntu from applying quotes around filenames and directory names
  • Thunderbird Attachment Download Error Corrupt Wrong filesize of 29 or 27 bytes Solution
  • Generic IP Camera LAN Default IP Settings DVR