Do you hate how Centos 7 defaults to allocating most of your valuable space to /home even though it is a production server?
Here is a quick guide on how to take back that space live, while online (of course make sure you have backups just in case something goes wrong!):
First we will reduce our home dir by 100G:
lvreduce -L -100G /dev/mapper/centos-home
WARNING: Reducing active and open logical volume to ........
In a RAID array I had a have periodically lost a drive here and there over the past several months. Iwas always able to readd and resync without losing data. However at some point it looks like some minor corruption happened and this makes DRBD unhappy.
Using fsck did not help either.
Dec 19 06:01:45 storageboxtest4 kernel: [19005.945890] EXT3-fs error (device drbd0): ext3_get_inode_loc: unable to read inode block - inode=22184379........
I've never seen this before in all of my years. Ihave some very old CDs and DVDs 12-15 years old that seem not to work in this BD-R/DVD-R/CD-R Asus drive.
The discs are fine actually and ironically they even work fine on a normal LG USB based BD-R drive!
Here are the errors in Linux:
[2914936.884924] attempt to access beyond end of device
[2914936.884927] loop1: rw=0, want=730424, limit=688384
[2914954.556873] attempt to........
user@box:~$ sudo tune2fs -l /dev/md99
[sudo] password for user:
tune2fs 1.42.9 (4-Feb-2014)
Filesystem volume name:
Last mounted on: /mnt/md50
Filesystem UUID: 976a8655-2619-4587-878c-dab07f7b7652
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
This is a 8TB Seagate external USB 3.0 device apparently newer kernels use a module called "UAS" instead of "USB Storage" which causes issues as a lot of devices are not properly supported in UAS mode by the kernel driver. The solution some say is to disable UAS specifically for your USB device but I'd rather just disable UAS altogether.
Solution blacklist UAS: *do not do this it does not work and just causes your USB 3.0........
This was a surprising bug but I unplugged all drives for an array md127. At first it was just 1 drive and mdadm seemed to notice this. I unplugged the second drive taking the array offline but mdadm did not realize it was offline and still showed a non-existent disk as being part of it. This created problems trying to unmount it or even to stop this array with mdadm freezing.
As for how to fix it I can only think of making sure you are not in a mounted path of........
This happened while an mdadm array was syncing, all access from writing a new blank file to opening a small .txt file was very slow:
[222117.312078] kjournald starting. Commit interval 5 seconds
[222117.685060] EXT3-fs (md0): using internal journal
[222117.685096] EXT3-fs (md0): mounted filesystem with ordered data mode
[222122.376847] kjournald starting. Commit interval 5 seconds
[222122.602825] EXT3-fs (md2): using internal jour........
Here is the scenario you or a client have a remote machine that was installed as a standard/default minimal Centos 6.x machine on a single disk with LVM for whatever reason. Often many people do not know how to install it to a RAID array so it is common to have this problem and why reinstall if you don't need to? In some cases on a remote system you can't easily reinstall without physical or KVM access.
So in this case you add a second physical or disk or already ha........
Use netstat with the -anpe option. The e option shows the inodes and I do not know if it will always work or if it was by fluke but I was dealing with dozens of SSHsessions and needed to know which session was related to which forward (the PIDs of the SSHand SSHD did not match etc...)
Notice the "59560675" and "59560762" those are almost identical, if you find two sets that are nearly identical except for the last 3 digits they may match (in my ca........
Sep 26 16:56:21 box kernel: 00 00 00 00 00 00 00 00
Sep 26 16:56:21 box kernel: [37007.155690] d_alias libdl-2.12.so d_count=9 d_flags=8
Sep 26 16:56:21 box kernel: [37007.155697] 09 00 00 00 08 00 00 00 9f 05 9f 05 00 00 00 00 c0 71 1d 18 04 88 ff ff 00 00 00 00 00 00 00 00 a0 7e 48 00 00 c9 ff ff 78 a9 21 18 04 88 ff ff 3a 7b fa 4e 0d 00 00 00 98 5c 2d 18 04 88 ff ff 18 5c 2d 18 04 88 ff ff 18 5c 2d 18 04 88 ff ff 00 01 10 00 00 00 ad de 00 02 20 00 00 00 ad de f8........
mkdir: cannot create directory 'test': Disk quota exceeded
You are out of inodes usually:
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/simfs 200000 200000 0 100% /
Jun 12 10:30:53 kernel: [724514.291670] EXT4-fs error (device md20): ext4_add_entry: bad entry in directory #2552670: rec_len % 4 != 0 - offset=216, inode=2553603, rec_len=94, name_len=84
Jun 12 10:30:53 kernel: [724514.292400] EXT4-fs error (device md20): ext4_add_entry: bad entry in directory #2552670: rec_len % 4 != 0 - offset=216, inode=2553603, rec_len=94, name_len=84
This happened when an unprivileged user was backing up a Windows filesystem, once they ha........
The best way I could figure out is to use another guest of some sort to do this, while assigning the disk that needs to be resized to the same guest.
So say we have /dev/xvda as the guests drive and we've booted it up.
We also have /dev/xvdb (this is going to be the image/disk to be resized).
In this case it's based on an ext3/4 image.
Run e2fsck on it to ensure there are no filesystem errors.
fdisk -lu VPS.img
last_lba(): I don't know how to handle files with mode 81ed
You must set cylinders.
You can do this from the extra functions menu.
Disk VPS.img: 0 MB, 0 bytes
255 heads, 63 sectors/track, 0 cylinders, total 0 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End ........
This is one in a series of weird things whichIthought was motherboard related (I RMA'd the motherboard), the RAM tests fine with memtest86 and I used badblocks on both RAID 1 members with no errors and smartctl is happy with them.
Basically the array crashes the kernel a lot and has issues when writing.
[112322.723465] md0: rw=0, want=14958668696, limit=1887460480
[112322.731077] attempt to access beyond end of device
These were caused by a bad stick of Corsair RAM
I like dd, although it only reads it, usually a read test of the entire disk will uncover if your hard drive is bad in some parts. This is a good thing to do at least once a month, a lot of times bizarre program behavior, laginess and crashing/unnmounting problems etc.. are due to a failing disc and SMART won't know it or indicate a problem:
We must also remember there's never a guarantee, I've found that ever since we moved to larger and more platters per drive with 1TB drives........
high IO wait
424 root 39 19 1900 848 552 D 0.0 0.0 0:00.91 updatedb
root 424 0.0 0.0 1900 848 ? DN Mar11 0:00 /usr/bin/updatedb -f sysfs?rootfs?bdev?proc?cpuset?binfmt_misc?debugfs?sockfs?usbfs?pipefs?anon_inodefs?futexfs?tmpfs?inotifyfs?eventp........
I think this will be useful to others because I have a server that kept crashing mysteriously during intense disk usage/RAID checks. It would only crash during the weekly RAID integrity check.
ThenI noticed during a reboot that not all CPUs were being brought up, as a result this actually creates much higher temperatures with the output I got from sensors, just booting the system produced higher than normal temperatures.
You can imagine that a full blown RAID check........
Openvz problem, it is confusing because it's an inode issue and there is enough free space.
cp: cannot create regular file `forums/memberlist.php': Disk quota exceeded
/dev/simfs 60G 20G 41G 33% /
none 2.0G 4.0K 2.0G 1% /dev........
Jan 16 04:02:03 centosbox syslogd 1.4.1: restart.
Jan 16 04:07:34 centosbox kernel: INFO: task updatedb:20771 blocked for more than 300 seconds.
Jan 16 04:07:34 centosbox kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 16 04:07:34 centosbox kernel: updatedb D F78BE050 6476 20771 20766&n........
Nov 29 20:17:58 ubuntu kernel: [ 1157.180789] md: md1 stopped.
Nov 29 20:17:58 ubuntu kernel: [ 1157.180829] md0: unknown partition table
filesystem not responding/reading properly with du or rsync (this needed an fsck).
[ 2571.489217] EXT3-fs error (device md2): ext3_lookup: deleted inode referenced: 35923106
[ 2571.942299] EXT3-fs error (device md2): ext3_lookup: deleted inode referenced: 35923110
Initializing quota ...
Error: Not enough parameters, diskinodes quota not set
vzctl set $veid --diskinodes 90000:91000 --save
New versions of OpenVZ seem to have some strange diskinodes parameter which is required.........
This is a great way to use your ftp server space, for example on your web hosting account (althoughI believe many hosts don't allow storage like this), but if you have a VPS/Dedicated Server etc.., this would be perfect. Imagine how easy it is to work with an ftp account that you can just mount as a normal partition or directory in Linux, it would be great for backups etc..
curlftpfs - mount a ftp host as a local directory
Before we start I take no responsibility for this, you should have a backup and if you make a mistake during this process you could wipe out all of your data. So backup somewhere else before starting this as a precaution, or make sure it's data you could afford to lose.
The RAID 1 Setup (Hardware Wise)
I've already setup my 2 x 1TB (Seagate) drives with identical partitions, make sure your new hard drive (the empty one) is setup like your curr........
I have no idea why but mkfs.ext3 defaults to a patheticlly small blocksize of 1024 bytes/1KB (kilobyte). That means the maximum filesize is ONLY 16GB! With 2KB/2048 bytes you get a 256 GB maximum filesize, and with 4KB/4096 bytes you get 2TB!
I finally noticed/paid attention to this after realizing that with rsync and scp that no file larger than 17GB could be transferred. I then realized it must be a file size limit on the partition.
Here is what tune2fs tol........
You can see the problem below, GRUB recognizes my hard drive and sees the partitions but cannot access them. This is from the GRUB boot disc I'm using.
What happened is that I had some power issues causing this system's power to be interrupted several times and basically an on/off on/off situation.
The system won't boot, I just get a flashing cursor and no message or error from GRUB. Obviously the problem is that my MBR seems corrupt or some other issue.
When trying to even cd or ls the mounted OCFS2 partition it crashes. Ithink this is a combination of VMWare Server's problem and the way I mounted and symlinked to it.
More than anything this shows the problem and lack of forsight with VMWare, but also that OCFS2 is easily crashed if you do strange things.
Output of /var/log/messages for OCFS2
Apr 10 15:57:45 localhost kernel: [84331.691258] Modules linked in: vmnet vmci vmmon ocfs2_stac........