mdadm RAID 1 adventures

I separated the 2 drives in the RAID 1 array.
1 is the old one /dev/sda and is out of date, while the separated other one /dev/sdc was in another drive and mounted and used with more data (updated).

I wonder how mdadm will handle this:

usb-storage: device scan complete
md: md127 stopped.
md: bind
md: md127: raid array is not clean -- starting background reconstruction
raid1: raid set md127 active with 1 out of 2 mirrors
md: md126 stopped.
md: bind
raid1: raid set md126 active with 1 out of 2 mirrors
md: md125 stopped.
md: bind
raid1: raid set md125 active with 1 out of 2 mirrors
kjournald starting. Commit interval 5 seconds
EXT3-fs warning: checktime reached, running e2fsck is recommended
EXT3 FS on md127, internal journal
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
spurious 8259A interrupt: IRQ7.
md: md127 stopped.
md: unbind
md: export_rdev(sda3)
ata1: exception Emask 0x10 SAct 0x0 SErr 0x90202 action 0xe frozen
ata1: irq_stat 0x00400000, PHY RDY changed
ata1: SError: { RecovComm Persist PHYRdyChg 10B8B }
ata1: hard resetting link
ata1: link is slow to respond, please be patient (ready=0)
ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata1.00: configured for UDMA/133
ata1: EH complete
sd 0:0:0:0: [sda] 3907029168 512-byte hardware sectors (2000399 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
ata3: exception Emask 0x10 SAct 0x0 SErr 0x40d0000 action 0xe frozen
ata3: irq_stat 0x00400040, connection status changed
ata3: SError: { PHYRdyChg CommWake 10B8B DevExch }
ata3: hard resetting link
ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata3.00: ATA-8: WDC WD20EARS-00S8B1, 80.00A80, max UDMA/133
ata3.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32)
ata3.00: configured for UDMA/133
ata3: EH complete
scsi 2:0:0:0: Direct-Access ATA WDC WD20EARS-00S 80.0 PQ: 0 ANSI: 5
sd 2:0:0:0: [sdc] 3907029168 512-byte hardware sectors (2000399 MB)
sd 2:0:0:0: [sdc] Write Protect is off
sd 2:0:0:0: [sdc] Mode Sense: 00 3a 00 00
sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 2:0:0:0: [sdc] 3907029168 512-byte hardware sectors (2000399 MB)
sd 2:0:0:0: [sdc] Write Protect is off
sd 2:0:0:0: [sdc] Mode Sense: 00 3a 00 00
sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sdc: sdc1 sdc3
sd 2:0:0:0: [sdc] Attached SCSI disk
sd 2:0:0:0: Attached scsi generic sg2 type 0

-bash-3.1# fdisk -l

Disk /dev/sda: 2000.3 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 1 1825 14659281 83 Linux
/dev/sda2 * 1826 9121 58605120 83 Linux
/dev/sda3 9122 243201 1880247600 83 Linux

Disk /dev/sdb: 1002 MB, 1002438656 bytes
255 heads, 63 sectors/track, 121 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 * 1 122 978912+ c W95 FAT32 (LBA)
Partition 1 has different physical/logical endings:
phys=(120, 254, 63) logical=(121, 222, 37)

Disk /dev/md126: 60.0 GB, 60011577344 bytes
2 heads, 4 sectors/track, 14651264 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md126 doesn't contain a valid partition table

Disk /dev/md125: 15.0 GB, 15009981440 bytes
2 heads, 4 sectors/track, 3664546 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md125 doesn't contain a valid partition table

Disk /dev/sdc: 2000.3 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdc1 * 1 1825 14659281 83 Linux
/dev/sdc3 9122 243201 1880247600 83 Linux
-bash-3.1# mdadm -A -s
mdadm: /dev/md/diaghost05102010:2 exists - ignoring
mdadm: /dev/md127 has been started with 1 drive (out of 2).
mdadm: /dev/md/diaghost05102010:2 exists - ignoring
mdadm: /dev/md124 has been started with 1 drive (out of 2).
-bash-3.1# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] [multipath]
md124 : active raid1 sda3[1]
1880246440 blocks super 1.2 [2/1] [_U]

md125 : active raid1 sda1[1]
14658185 blocks super 1.2 [2/1] [_U]

md126 : active raid1 sda2[1]
58605056 blocks [2/1] [_U]

md127 : active raid1 sdc3[0]
1880246440 blocks super 1.2 [2/1] [U_]

unused devices:

dmesg:

md: md127 stopped.
md: bind
md: bind
md: kicking non-fresh sda3 from array!
md: unbind
md: export_rdev(sda3)
raid1: raid set md127 active with 1 out of 2 mirrors
md: md124 stopped.
md: bind
raid1: raid set md124 active with 1 out of 2 mirrors

=====================
It seems it considers the new sdc (up to date) disc as /dev/md124
The old one is /dev/md127

So it somehow thinks they are totally separate arrays.

mdadm -D /dev/md127
/dev/md127:
Version : 1.2
Creation Time : Mon May 10 22:05:48 2010
Raid Level : raid1
Array Size : 1880246440 (1793.14 GiB 1925.37 GB)
Used Dev Size : 1880246440 (1793.14 GiB 1925.37 GB)
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent

Update Time : Thu Dec 2 18:46:10 2010
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0

Name : diaghost05102010:2
UUID : d851091d:3d109a17:921ab0e1:3a465899
Events : 89394

Number Major Minor RaidDevice State
0 8 35 0 active sync /dev/sdc3
1 0 0 1 removed

===========
mdadm -D /dev/md124
/dev/md124:
Version : 1.2
Creation Time : Mon May 10 22:05:48 2010
Raid Level : raid1
Array Size : 1880246440 (1793.14 GiB 1925.37 GB)
Used Dev Size : 1880246440 (1793.14 GiB 1925.37 GB)
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent

Update Time : Fri Dec 3 02:34:08 2010
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0

Name : diaghost05102010:2
UUID : d851091d:3d109a17:921ab0e1:3a465899
Events : 38

Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 3 1 active sync /dev/sda3



Actually this seems smart, it seems to know that /dev/sdc3 is the fresh one.
I don't want to test this, but I wonder what would have happened if I had md127 mounted before using just sda3, how would mdadm react if it was part of a live filesystem? I'm guessing (hopefully) it wouldn't allow sdc3 to join the array and let the user decide how to handle it.

This could still be an issue, why did mdadm when I did a -A -s find md124 as a new array when the UUID for both is the same?
I kind of get it, it sees md124 as being the other out of sync array.

I'm going to manually kill md124

mdadm --manage --stop /dev/md124

Now let's add the out of sync/old one back manually to the array.

mdadm --add /dev/md127 /dev/sda3
mdadm: re-added /dev/sda3
-bash-3.1# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] [multipath]
md125 : active raid1 sda1[1]
14658185 blocks super 1.2 [2/1] [_U]

md126 : active raid1 sda2[1]
58605056 blocks [2/1] [_U]

md127 : active raid1 sda3[1] sdc3[0]
1880246440 blocks super 1.2 [2/1] [U_]
[>....................] recovery = 0.0% (210496/1880246440) finish=446.5min speed=70165K/sec



good it worked as it should, when checking mdadm it clearly sees sdc3 as the active device and sda3 as the spare (I wish the /proc/mdstat would clearly show which device is active and which one is being synced to):



mdadm -E /dev/sda3
/dev/sda3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : d851091d:3d109a17:921ab0e1:3a465899
Name : diaghost05102010:2
Creation Time : Mon May 10 22:05:48 2010
Raid Level : raid1
Raid Devices : 2

Avail Dev Size : 3760493152 (1793.14 GiB 1925.37 GB)
Array Size : 3760492880 (1793.14 GiB 1925.37 GB)
Used Dev Size : 3760492880 (1793.14 GiB 1925.37 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 3f924621:bd0d3b67:917ec834:90ef479f

Update Time : Fri Dec 3 03:01:21 2010
Checksum : 4acbcec8 - correct
Events : 89460


Device Role : spare
Array State : A. ('A' == active, '.' == missing)
-bash-3.1# mdadm -E /dev/sdc3
/dev/sdc3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : d851091d:3d109a17:921ab0e1:3a465899
Name : diaghost05102010:2
Creation Time : Mon May 10 22:05:48 2010
Raid Level : raid1
Raid Devices : 2

Avail Dev Size : 3760493152 (1793.14 GiB 1925.37 GB)
Array Size : 3760492880 (1793.14 GiB 1925.37 GB)
Used Dev Size : 3760492880 (1793.14 GiB 1925.37 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 5302b74a:fbb3097e:5f960dc5:b2d1461a

Update Time : Fri Dec 3 03:01:21 2010
Checksum : 964edf09 - correct
Events : 89460


Device Role : Active device 0
Array State : A. ('A' == active, '.' == missing)
-bash-3.1#



So today we learned that mdadm isn't as smart as we think, it shouldn't have created two separate arrays when the UUID's matched.
It should have just kicked the unfresh device from the array like it did but not create a separate array.

It also isn't as smart as say DRBD (yes I know it's network block replication) but it's basically RAID 1 over the network and doesn't sync the entire disc contents. I think it would be interesting if the DRBD method of marking blocks as being out of sync could apply to mdadm. It is rather silly that if there are any changes to the disk, it has to be entirely rebuilt, whereas DRBD would just rebuild the changed contents.


Tags:

mdadm, raid, adventuresi, array, dev, sda, sdc, mounted, updated, usb, scan, md, bind, reconstruction, active, mirrors, kjournald, interval, ext, fs, checktime, fsck, recommended, filesystem, mode, spurious, irq, unbind, export_rdev, ata, exception, emask, sact, serr, xe, irq_stat, phy, rdy, serror, recovcomm, persist, phyrdychg, resetting, sata, gbps, sstatus, scontrol, configured, udma, byte, hardware, sectors, mb, cache, enabled, doesn, dpo, fua, commwake, devexch, wdc, wd, multi, lba, ncq, depth, scsi, pq, ansi, disk, generic, sg, bash, fdisk, gb, bytes, cylinders, linux, sdb, partition, endings, phys, contain, valid, diaghost, ignoring, proc, mdstat, personalities, linear, multipath, _u, u_, unused, devices, dmesg, considers, disc, arrays, creation, gib, persistence, superblock, persistent, update, thu, dec, degraded, uuid, ab, raiddevice, sync, fri, wouldn, user, manually, min, synced, efc, feature, avail, offset, bd, ec, ef, checksum, acbcec, fbb, dc, edf, isn, shouldn, matched, unfresh, drbd, replication, contents, method, marking, rebuilt, whereas, rebuild,

Latest Articles

  • apache symlinks denied even with followsymlinks
  • chown how to change ownership on a symlink
  • how to use ifplugd in Linux to execute a command or script when a NIC cable is unplugged or plugged in
  • dd how to backup and restore disk images including compression with gzip
  • mpv / mplayer with Radeon / AMD GPU Video Card Driver enable VDPAU GPU Accelerated Video Decoding
  • Wordpress Reset Blog User Password from MySQL Using Linux Bash and not PHPMyadmin
  • Ubuntu Linux Mint Debian xorg performance and tear-free tuning for AMD Radeon Based Cards
  • Centos 7 Stopped and Disabled Firewalld and ports still blocked
  • MariaDB / MySQL Reset Root Forgotten Password on Centos 7
  • Centos 7 How to install Mysql/Mariadb
  • PHP 7.2, Apache and Centos 7 How To Install
  • Ubuntu Debian Linux Mint r8169 r8168 Network Driver Problem and Solution
  • Linux 3D Performance benchmarks with glxgears 59-60fps solution
  • How To Install Asterisk 16 17 on Debian Ubuntu Linux
  • Linux Ubuntu Debian Centos How To Make a Bootable Windows 7, 8, 10, 2016, 2019 Server USB from ISO
  • How To Restore Windows MBR Bootsector from Linux using syslinux
  • Linux Ubuntu Cannot Print Large Images
  • Cannot Print PDF Solution and Howto Resize
  • Linux Console Login Screen TTY Change Message
  • Apache Cannot Start Listening Already on 0.0.0.0