I'm looking for a little help re-gaining access to an LVM2 area on a spindle that came out of a crippled machine. Google has stopped helping me so I thought I'd ask for advice. The system was running ubuntu 8.04 LTS and I configured it using two pairs of disks with MD RAID-1 mirrored partitions. Filesystems for /boot and / were regular ext3 on MD. The rest of the system was LVM2 on top of MD.
I used amanda to back up to tape for most backups. I have the amanda files in my "offsite" format on disk from a couple of weeks before things went haywire so I'm not totally screwed but I'd really like access to the latest versions of the files from disk if I can. Using my daily tapes with no working server (running amanda) could be possible but may be more trouble than it's worth. Right now I have the spindle with desired data connected to another old desktop booting an ubuntu rescue.
I'm a little puzzled about what happened. The two spindles with /, /boot, /home, /var and /project (a place where I stuffed working files requiring multi-user access) were on separate power rails, separate PATA (yes, PATA) controllers etc... A hardware issue put a crater in one of the ICs on the first spindle and then the system refused to finish booting from then on. GRUB starts to boot but all of the MD devices were no longer "running". The partitions are still there. If I mount /boot and / as read-only EXT3 the data is there. It looks like all of the MD metadata is gone. The UUIDs are all zeros and the MD arrays cannot be detected with "mdadm --examine --scan ....". Is this what it would look like if at the time of the failure the array was resilvering and what is now the remaining good device was considered out of sync? or did something else happen?
As mentioned, I can access /boot and / by simply mounting the MD member partitions as ext3. I cannot, however, get LVM2 to recognize anything on the remaining large partition. I changed partition type from fd to 8e (Linux LVM) and still nothing. I dd'ed the first few sectors of the partition and in the plain text portions I can see LVM2 metadata.
What's my best course of action here?
Thanks,
I used amanda to back up to tape for most backups. I have the amanda files in my "offsite" format on disk from a couple of weeks before things went haywire so I'm not totally screwed but I'd really like access to the latest versions of the files from disk if I can. Using my daily tapes with no working server (running amanda) could be possible but may be more trouble than it's worth. Right now I have the spindle with desired data connected to another old desktop booting an ubuntu rescue.
I'm a little puzzled about what happened. The two spindles with /, /boot, /home, /var and /project (a place where I stuffed working files requiring multi-user access) were on separate power rails, separate PATA (yes, PATA) controllers etc... A hardware issue put a crater in one of the ICs on the first spindle and then the system refused to finish booting from then on. GRUB starts to boot but all of the MD devices were no longer "running". The partitions are still there. If I mount /boot and / as read-only EXT3 the data is there. It looks like all of the MD metadata is gone. The UUIDs are all zeros and the MD arrays cannot be detected with "mdadm --examine --scan ....". Is this what it would look like if at the time of the failure the array was resilvering and what is now the remaining good device was considered out of sync? or did something else happen?
As mentioned, I can access /boot and / by simply mounting the MD member partitions as ext3. I cannot, however, get LVM2 to recognize anything on the remaining large partition. I changed partition type from fd to 8e (Linux LVM) and still nothing. I dd'ed the first few sectors of the partition and in the plain text portions I can see LVM2 metadata.
What's my best course of action here?
- Should I force MD to build another RAID and consider this remaining good partition to be a member while the other is failed? Doesn't this risk overwriting good data?
- Is there some lvm2 foo I'm missing to use that remaining MD partition without running MD any longer? I know that MD and ext3/4 were designed not to interfere with one another so a broken MD mirror could simply be mounted as ext3/4. I'm not sure if LVM2 and MD play as nicely or whether the MD metadata confuses LVM2 which is why it's not working
- I have read http://www.linuxjournal.com/article/8874 The problem is that none of the metadata is found using mdadm --examine or vgchange -a y
- any idea why a failure of one disk caused all my UUIDs and metadata to be lost? I tested all this stuff when I built the box but a real failure left me hanging.
Thanks,