Thursday, February 17, 2005
Solaris Metadevices--
Last night at about eight I got a phone call from our data manager. She'd
rebooted the main database server and it failed to boot ...
It turned out when I got here that the reason the boot had failed was that
it couldn't find its RAID array. Fortunately the system boots off a set of
disk mirrors, the RAID array is only used for storing the database, so we
were sitting at the infamous "enter the root password for maintenance"
prompt.
Once I got in, I discovered what was wrong with the RAID array. A
metastat
-i and a metadb
-i both produced the same result:
an error message to the effect that there were no databases configured.
This seemed a bit odd to me as /etc/lvm/md.cf and
/etc/lvm/mddb.cf both existed and seemed to contain
"useful" information. Eventually after a hour or so of soul
searching, RTFMing, attempting things like metadb
-p, and generally
worrying, I came to the conclusion that the metadevice state database had
well and truly vanished and that the information in mddb.cf
was a load of bollocks.
This was when I discovered that the machine had no md.tab
file. This is probably a remnant of our previous disaster with this machine, but was
nevertheless some cause for concern.
Fortunately, for the second time, Big
Brother came to the rescue. One of the things tucked away in its
historical logs was a copy of the metadb
-i output for the machine. Kudos to Galen Johnson & friends for
their metadevice
extension to BB4.
After working all of this out, recovery was a matter of creating a correct
md.tab
file, reinitialising the metadevice state databases from this file, and then
reinitialising the metadevices themselves. The biggest problem in this was
trying not to make typos in the many variants of c1t2d0s1 that
needed to be typed in the correct order.
Moral of the story. Make sure you have an md.tab file (you can
create most of one with metastat -p > md.tab, you just need to add
your own mddbXX lines based on the output of metadb -i). Make sure
this file is well backed up, or preferably printed out. Have lots of
coffee, patience, and someone to check your typos.
posted by guy at: 09:10 SAST |
path: /systems |
permanent link
