[gentoo-user] Oh no! My raid5 is not assembled, and how I fixed it.

I rebooted to upgrade to kernel 3.4.1. I accidentally had the
combination of uvesafb, nouveau kms and nvidia-drivers enabled, which
caused my system to go blank after rebooting. I was not able to SSH
into the machine, so I did the magic-sysrq REISUB to reboot into my
previous kernel. When it booted into the previous kernel (3.3.5), I
saw a whole bunch of "I/O error" messages scrolling by, for every disk
in my RAID array. I have never seen these errors before. I hoped it
was just some module confusion because I was booting a different
kernel. I was able to boot into my root filesystem, but the raid did
not assemble. After blacklisting nouveau and rebooting into 3.4.1,
there were none of the I/O errors mentioned, but mdraid failed with
this message:

* Starting up RAID devices ...
* mdadm main: failed to get exclusive lock on mapfile
mdadm: /dev/md2 is already in use.
mdadm: /dev/md1 is already in use.
[ !! ]

Oh no! Heart beating quickly... terabytes of data... Google finds
nothing useful with these messages.

My mdadm.conf has not changed, no physical disks have been added or
removed in over a year. mdadm configuration has not changed at all. I
have of course updated hundreds of packages since my last reboot,
including mdadm.

From the /proc/mdstat it shows that it's not detecting all of the
member disks/partitions:

Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5]
[raid4] [multipath] [faulty]
md1 : inactive sdb1[0](S)
1048575868 blocks super 1.1

md2 : inactive sdf2[5](S)
904938415 blocks super 1.1

unused devices: <none>


Those normally included all disks in sdb through sdf, partition 1 and
2 from each disk.

My mdadm.conf has always had only two ARRAY lines (for /dev/md1 and
/dev/md2) with the UUID of the arrays. Previously the member disks
were always automatically detected and assembled when I booted and
started mdadm. Running mdadm --query --examine on the partitions
showed they did still contain the valid raid information. So I felt
confident in trying to reassemble it.

To fix, I did:

/etc/init.d/mdraid stop

to stop the array (could have also done "mdadm -Ss", which is what the
stop script did)

Then I edited mdadm.conf and added a device line:

DEVICE /dev/sd[bcdef][12]

So now I am telling it specifically where to look. I then restarted mdraid:

/etc/init.d/mdraid start

et voilà! my raid was back and functioning. I don't know if this is a
result of a change in kernel or mdadm behavior, or simply a result of
my REISUB that left the raid in a strange state.