I've had tons of raid failures in my main server the last couple of months (All in all 6 disks I think that have failed fairly recently). I yesterday realized that perhaps the problem was the way I had run the wires inside the box. Once I opened it I realized I had squeezed the SATA wired together with power cables that had pretty high amps (After all they were powering 17 disks and a bunch of fans). What basically happened was that with alarming regularity I hat the drives time out and just completely lock up until I rebooted (Causing them to have been failed from the RAID by that time). Then after a reboot they usually worked fine for a a couple of days until it happened again.
It was also fairly obvious that certain hotswap slots were more prone to fail in this way than others. So basically now I have rewired the box so that the power cables go over the top of the fans in the middle of the box and the SATA cables go below it and so far it seems to work out fine. For the first time in ages I am running with completely synced raids in all my servers.
[pallas]$ cat /proc/mdstat Personalities : [raid6] md0 : active raid6 sda1 sdg1 sdm1 sdo1 sdp1 sdl1 sdk1 sdj1 sdi1 sdh1 sdn1 sdf1 sde1 sdd1 sdc1 sdb1 6837375104 blocks level 6, 64k chunk, algorithm 2 [16/16] [UUUUUUUUUUUUUUUU] unused devices:
[cadiz]$ cat /proc/mdstat Personalities : [raid6] md0 : active raid6 sdp1 hdc1 sdq1 sdr1 sdo1 sdn1 sdm1 sdl1 sdk1 sdj1 sdt1 sdi1 sde1 sdf1 sdg1 sdh1 sdd1 sdc1 sds1 sdu1 sdb1 sda1 hda1 5128113984 blocks level 6, 64k chunk, algorithm 2 [23/23] [UUUUUUUUUUUUUUUUUUUUUUU] unused devices: [valdez]$ cat /proc/mdstat Personalities : [raid6] md1 : active raid6 hda2 sda2 sdc2 sdb2 sdd2 sdh2 sde2 sdf2 sdg2 hdb2 1789768704 blocks level 6, 64k chunk, algorithm 2 [10/10] [UUUUUUUUUU] md0 : active raid6 sdg1 sda1 sdc1 sdb1 sdd1 sdh1 sdf1 102373440 blocks level 6, 64k chunk, algorithm 2 [7/7] [UUUUUUU] unused devices:
I leave it up to the reader to figure out the total size of that read from that readout. Running with degraded raids is one of the few things that can really stress me out because I know if I lost the stuff I have on those servers it would be impossible to replace.
To end with a quote (This time by myself): Peace of mind is a synced raid