gwax's rumblings

There is a right way and there are wrong ways to add dri­ves to a RAID5 array with mdadm. Annoyingly, I chose one of the wrong ways last week when I went to increase the hard drive space in thevoid. Thankfully, my mis­take has proven more both­er­some and time con­sum­ing than harmful.

Starting out last week, thevoid had 4x 500GB dri­ves in an mdadm RAID5 con­fig­u­ra­tion: /dev/hde1, /dev/hdg1, /dev/sda1, /dev/sdb1. This had me sit­ting on 1.5TB of space with one drive worth of fail­abil­ity, which was prov­ing insuf­fi­cient. In order to resolve my space issues, I ordered 3x 500GB dri­ves and pro­ceeded to install them when the arrived: /dev/hdb, /dev/hdc, /dev/hdd.

In my haste to have more space, I pro­ceeded to add them to the array and grow it:

mdadm --manage /dev/md1 --add /dev/hdb
mdadm --manage /dev/md1 --add /dev/hdc
mdadm --manage /dev/md1 --add /dev/hdd
mdadm --grow /dev/md1 --raid-devices=7

This process then got to work and took about 3 – 4 days to com­plete. I assume the time was on account of most of the dri­ves being IDE dri­ves, many shar­ing chan­nels and there being an awful lot of space. Once the grow­ing process was com­plete, I merely needed to resize the ext3 par­ti­tion on the array:

resize2fs /dev/md1

This then took an hour or so and I was sit­ting on 3TB of total space. The only part of the process dur­ing which my drive was inac­ces­si­ble was when I had to turn off the com­puter to phys­i­cally install my IDE drives.

Everything is per­fect now, right? Wrong. Guess who for­got that you should par­ti­tion dri­ves before using them? That’s right, me. I really wanted to put a linux raid autode­tect par­ti­tion on each of the dri­ves before adding them to the array. Not that it really does much harm to add the dri­ves straight to the array but it’s poor form and it might pose prob­lems that I am not aware of in the future so, clearly, it’s a thing that should be fixed. Thankfully, being RAID5, my array is able to lose dri­ves and still be fine, thus allow­ing the solu­tion of fail­ing, remov­ing, par­ti­tion­ing and adding each of the new dri­ves back into the array:

mdadm --manage /dev/md1 --fail /dev/hdb
mdadm --manage /dev/md1 --remove /dev/hdb

par­tion /dev/hdb and then:

mdadm --manage /dev/md1 --add /dev/hdb1

now wait until the array has rebuilt itself:

cat /proc/mdstat

and repeat for the other dri­ves. Overall, it’s not hard but it’s annoy­ing and it does take about 1/2 to 2/3 of a day per drive to fix. Thankfully, though, now the endeavor is done and my array works cor­rectly with no lost data and min­i­mal down time. Hooray for soft­ware RAID and hav­ing 3TB in a sin­gle place.

Oh, and you don’t want to for­get to update /etc/mdadm.conf after every step of the process of you could have some poten­tial problems.