need help badly with raid5 recovery

[bonnyjliss]

i'm a programmer and not a IT guy.

i was asked to help a company my software company is partnered with (small business, no IT dept)

they had a guy configure a linux mdadm raid5 with 4 drives. one drive failed at some point (it still does click of death). at some point someone there removed the 5th drive that was getting backups put on it. their last backup is from almost 3 years ago

they had it booting off a USB which became corrupted somehow and it would no longer boot, they had a non IT guy there try to install linux, i dont know exactly what happened but the third drive won't assemble into the raid array anymore. it keeps saying it can't find a superblock, it has a linux (bootable) ext4 partition, which is different than the two good drives.

at the point this got put in my lap the two good drives have had the partitions written over, i'm using testdisk to recover the old deleted partitions but it is taking a long time to scan i'm waiting for it now...

the "good" drive that has the linux "bootable" partition on it, i've backed up with DD. i'm going to back up the other two drives with DD but it took 40 hours to do the first one

if i try to assemble the raid array with "assume clean" does it need the partitions sorted out? can mdadm figure it out?

i'm learning as i go on some of this .

i think they are probably f'd and might have to send it to some company to try and recover stuff forensically

 

[auntjemima]

Yeaahhhhh I am pretty sure its fucked, bud.

One drive completely useless and another missing? I would say you're up shit creek without a paddle.

 

[Krispy Kritter]

I would agree with that assessment. You can lose a single drive and that's it (with a 4 drive array). And to top it all off, they have been mucking about with the drives. Gonna waste a lot of time and money for likely nothing. I'd suggest they spend that money on new hardware as needed, ditch raid, and just start moving on.

 

[Zedicus]

the guy that tried to install linux ran the install onto one of the MDADM RAID disks. that coupled with the fact that one of the other drives is bad meant that you have 2 drives of a 4 drive raid 5 array. even if you were able to do a recovery on the drive that has the ext4 partition table on it now and manage to restore the MDADM RAID table, there would be no comparable hash for the RAID 5 so all or some or maybe none of the data would be no good but there would be no way to tell with out manual verification of EVERYTHING.

this is a sad story of a good idea being slowly ruined by transition through multiple people.

 

[mwroobel]

... And another sad realization that RAID is not a backup!

 

[kdh]

There is no more data. Its gone. =( You can't recover from that.

If you are serious about recovering anything, you'll stop touching it and reach out to a professional service to do it.

 

[Shoganai]

RAID needs backup as well. I would highly advise whoever is in charge to seek help from a professional. Although at this point, it may not be enough.

 

[Jandor]

Well, this proves that raid 5 is a no go compared to raid 1 who has plenty of advantages. You don't lose that much more data space by using raid 1 instead of raid 5, when you can go with 16TB HD. Those were the old days when raid 5 meant something. Then for super proofness you can put a third drive for spare and don't forget the regular backups.

Using raid1 on all PC I assemble for me and friends for more than 20 years. First on Linux since around 95, and later used old hack to activate raid 1 software on Windows 2000 Pro (mirror that could even be made easily bootable in case of failure). So since then all Desktops and gaming were raid 1. Then went only with motherboards with in BIOS raid 1 feature from Intel or AMD. First Raid chipset I used was nvidia chipset for AMD CPUs.
Never had to cope with disk failure since then. I now use raid 1 on SSD, using 2 different brands and models (mainly Samsung and Crucial)

 

[Blue Fox]

Well, this proves that raid 5 is a no go compared to raid 1 who has plenty of advantages. You don't lose that much more data space by using raid 1 instead of raid 5, when you can go with 16TB HD. Those were the old days when raid 5 meant something. Then for super proofness you can put a third drive for spare and don't forget the regular backups.

Using raid1 on all PC I assemble for me and friends for more than 20 years. First on Linux since around 95, and later used old hack to activate raid 1 software on Windows 2000 Pro (mirror that could even be made easily bootable in case of failure). So since then all Desktops and gaming were raid 1. Then went only with motherboards with in BIOS raid 1 feature from Intel or AMD. First Raid chipset I used was nvidia chipset for AMD CPUs.
Never had to cope with disk failure since then. I now use raid 1 on SSD, using 2 different brands and models (mainly Samsung and Crucial)

The only thing if proves is how important backups are. RAID1 isn't going to save you with multiple disk failures, filesystem issues, or your PSU taking out everything. It's not some magic fix and efficiency is only similar with the smallest array size. I'm not going to try and recreate the multiple 150TB arrays I have using RAID1 or nested versions of it.

 

[Jandor]

RAID needs backup as well. I would highly advise whoever is in charge to seek help from a professional. Although at this point, it may not be enough.

Yes. Lost data cannot be rediscovered from void.
Only future proof setup is at stake, so that it doesn't happen again.
Frankly I had that problem with a small copmpany I helped. the guy inside who take care of the data and the people using the computers did everything to bring viruses and to crash the systems. I've been called on rescue several times.
So what I finally did is completly render all the computers accessible only on basic user level and I diminished the user rights to strict file access for current work. Internet super-filterd. Only one PC to grap and send the company mail. The guys were all raging at me and that made me laugh at them. They told the CEO they couldn't work in those conditions but I proved him they were wrong, trying to lie to him, not focusing on their job during work hours. One guy I found personal data on the computer that proved he was working on something else. I got him fired. guys complaining I didn't leave them having their personal private mail as french law force emplyers to. I told them I didn't care about the french law (what about if they don't have a mail at all... the law doesn't tell). I told the CEO that in case of huge problem he could look into the different power user and administration passwords for some other specialist if he needs to in case not freetime to look into it. I put each password in an envelope for each computer and part of the intranet in a desk HE could only access. All computers would have the same user account, same password so nobody would be allowed to put private data into it. In a bigger company I would have made one password for each computer with CEO able to access all of them from his computer... still not private use.
It's been 15 years and those computers and new ones I assembled nearly 10 years ago are still working like 10 hours/day without any need for administration since then. There is a new guy who makes nothing more than maintain and replicate what I did. I even advised him not to switch on Windows 10 as they don't need any new hardware for now.

 

[Jandor]

The only thing if proves is how important backups are. RAID1 isn't going to save you with multiple disk failures, filesystem issues, or your PSU taking out everything. It's not some magic fix and efficiency is only similar with the smallest array size. I'm not going to try and recreate the multiple 150TB arrays I have using RAID1 or nested versions of it.

150TB of data is not that sort of setup mentioned in the thread. So my rule applies in that situation. Raid 1 is safer, and much much easier to manage. Would this have been a raid 1 setup, it wouldn't have any such problem in case the raid setup might have been broken. In his case the problem occured at first because the raid setup has been broken and raid 5 recover is complicated. You really need to avoid any wrong move and also you cannot recreate easily your data by reading each disk (there are tools). All this is avoided by Raid 1 setups. They are a no-brainer.

 

[pendragon1]

Yeaahhhhh I am pretty sure its fucked, bud.

One drive completely useless and another missing? I would say you're up shit creek without a paddle.

There is no more data. Its gone. =( You can't recover from that.

If you are serious about recovering anything, you'll stop touching it and reach out to a professional service to do it.

that^^ in all honesty, if you have to ask for help here its way beyond what you can do, its way beyond me too. "im sorry, this is way beyond me" theres no shame in it. seek a pro or move on with proper backups in place.

 

[Jandor]

Try this : R-Studio. Runs on Windows. I used it very long time ago, more 15 years ago, was called R-Tools, for a data recovery on a voluntarily linux raid 1 setup disassembled and boot erased by some guy. I restored all the data, but that data was still on it. Not sure how you configured your raid 5 setup and how the reaming disk was erased, maybe only half erased... that tool could try to read. In principle it tries to read, does not write anything. I also could read half of a dying hard drive. Software will try the best without hanging by inability of the OS to recognize the format of the drive.

https://www.r-studio.com/

Kind of last resort try for rescue. A professional could try to read and reconstruct on a reformatted drive unerased data... but it will cost a lot. Not sure R-Studio doesn't take care of some of those things. The guy/company should be very experimented at this. There will be much work and maybe hours spent into looking into usable the raw data and how to virtually assemble parts of it maybe obtaining most of the data as half deleted files, unusable.

Not sure if there are better tools out there.

 

[steves06]

Try this : R-Studio. Runs on Windows ...

Stellar may also do it: https://www.stellarinfo.com/windows-raid-recovery.php

I used one of their tools for Linux recovery a while ago and got decent results. It wasn't RAID though so I can't vouch for that part of their suite. Some of them will give you a time period where you can get a refund if their tool won't do the job.

 

[David-Duc]

150TB of data is not that sort of setup mentioned in the thread. So my rule applies in that situation. Raid 1 is safer, and much much easier to manage. Would this have been a raid 1 setup, it wouldn't have any such problem in case the raid setup might have been broken. In his case the problem occured at first because the raid setup has been broken and raid 5 recover is complicated. You really need to avoid any wrong move and also you cannot recreate easily your data by reading each disk (there are tools). All this is avoided by Raid 1 setups. They are a no-brainer.

How do you create a 150 TB RAID1 volume? The largest HDD on EARTH right now is smaller than 20 TB

 

[Jandor]

How do you create a 150 TB RAID1 volume? The largest HDD on EARTH right now is smaller than 20 TB

Not me.
I kind of thought the same thing but... let go :

The only thing if proves is how important backups are. RAID1 isn't going to save you with multiple disk failures, filesystem issues, or your PSU taking out everything. It's not some magic fix and efficiency is only similar with the smallest array size. I'm not going to try and recreate the multiple 150TB arrays I have using RAID1 or nested versions of it.

So my answer was related to that post.

 

[Denpepe]

How do you create a 150 TB RAID1 volume? The largest HDD on EARTH right now is smaller than 20 TB

He uses multiple disks ofc to make a raid volume that big also, the largest SSD is 100TB https://nimbusdata.com/press/nimbus...e-100-terabytes-power-data-driven-innovation/ or maybe even more in the meantime

 

[thesmokingman]

i'm a programmer and not a IT guy.

i was asked to help a company my software company is partnered with (small business, no IT dept)

...

i think they are probably f'd and might have to send it to some company to try and recover stuff forensically

That sucks to be you. I hope this company has learned its lesson, but it seems they have not. That's why they didn't bring ppl experienced in this area. If I were you I would not want to be responsible for this cluster fuck. It's not fair to you either cuz there is no way of recovering what's been lost. Dude, 3 years no back ups, they deserve their fate! A decade ago studies showed that over 50% of small businesses die due to IT failures, ie. no backups. This is business darwinism.

And fyi, there's no recovering the data. You can pay to have bits pulled off the platter but that's useless because you still have to put the bits together at the end of it all and that's the problem. Raid reconstructor or other raid recovery... is still no guarantee and its a very painfully slow process and maybe, maybe it works but mostly it doesn't.

 

[Jandor]

He uses multiple disks ofc to make a raid volume that big also, the largest SSD is 100TB https://nimbusdata.com/press/nimbus...e-100-terabytes-power-data-driven-innovation/ or maybe even more in the meantime

bonnyjliss problem may not be in that range of data, probably something like Raid 5 made of 4 4GB disks (it's actually on the second photo) which means 12GB of storage. This is why I was suggesting going for 2 bigger disks of 12GB on raid 1 or two times 2 disks of 6GB on raid 1 instead of 4 on raid 5. Much more secure and easy to manage.

 

[danswartz]

I remember reading about a medium/small company that religiously backed up their business stuff to tape (this was some years ago). No IT person (of course), so some random bro was doing it. Whoever set up the tape HW/SW hadn't tested properly, and the tape backups were garbage. Then, they had a 2-disk failure on their RAID5 array. Tried to restore from backups. Backups are garbage. They tried every one of the dozen or so backup tapes. No joy. They had gone paperless, so they literally had no idea what their receivables or payables or anything else was. A couple of weeks later, they were out of business.

 

[Master_shake_]

i'm a programmer and not a IT guy.

i was asked to help a company my software company is partnered with (small business, no IT dept)

they had a guy configure a linux mdadm raid5 with 4 drives. one drive failed at some point (it still does click of death). at some point someone there removed the 5th drive that was getting backups put on it. their last backup is from almost 3 years ago View attachment 217609

they had it booting off a USB which became corrupted somehow and it would no longer boot, they had a non IT guy there try to install linux, i dont know exactly what happened but the third drive won't assemble into the raid array anymore. it keeps saying it can't find a superblock, it has a linux (bootable) ext4 partition, which is different than the two good drives.

at the point this got put in my lap the two good drives have had the partitions written over, i'm using testdisk to recover the old deleted partitions but it is taking a long time to scan i'm waiting for it now...

the "good" drive that has the linux "bootable" partition on it, i've backed up with DD. i'm going to back up the other two drives with DD but it took 40 hours to do the first one View attachment 217610

if i try to assemble the raid array with "assume clean" does it need the partitions sorted out? can mdadm figure it out?

i'm learning as i go on some of this .

i think they are probably f'd and might have to send it to some company to try and recover stuff forensically

View attachment 217611

View attachment 217612

pack it all up and send it to a recovery team and pray.

 

[Jandor]

pack it all up and send it to a recovery team and pray.

He could give a try to the software I mentioned. R-Studio. It's not going to kill his HDs, only try to read them and extract the data. Ithink it's around $50. Not a big deal even if it doesn't work in that case. Might be useful in other situations. It reads data from about anything.

 

[kdh]

150TB of data is not that sort of setup mentioned in the thread. So my rule applies in that situation. Raid 1 is safer, and much much easier to manage. Would this have been a raid 1 setup, it wouldn't have any such problem in case the raid setup might have been broken. In his case the problem occured at first because the raid setup has been broken and raid 5 recover is complicated. You really need to avoid any wrong move and also you cannot recreate easily your data by reading each disk (there are tools). All this is avoided by Raid 1 setups. They are a no-brainer.

I personally manage an 8p FC environment right now that does over a million iops. No raid type is better then the other and no raid type is more complicated then the other when you actually know how to set them up and configure them. It depends on the work load, drive type, storage platform and budget. Raid1 maybe be fine in couple of T mom and pop shop on its a chumpy 4 port controller. But absolutely no real pro shop only does raid1 outside of that because its "safer". Yah, no.. Doesn't work that way.

Technically, raid 6 is safer then raid 1 with is dual fault failure tolerance..

Raid1 wouldnt have fixed ops original problem as it was already double faulted when he got it. Raid1 wouldn't have saved that. Backup volume missing, and then someone else over wrote one of the drives? They had bigger problems from day 1 then a raid issue.

 

[kdh]

How do you create a 150 TB RAID1 volume? The largest HDD on EARTH right now is smaller than 20 TB

I backup Ps of space nightly.. its called dedupe and compression, I use avamar, datadomain and isilon. My previous gig, I backed up 65TB+ oracle DBs on the regular over to isilon or a monster tape robot.

edit.. one more bit.. storage expanded to pools from raid. You build raid, then build pools out of raid groups. Been around for 10 years at this point. I have stroage pools well overr 500TBs.

 

[Jandor]

I personally manage an 8p FC environment right now that does over a million iops. No raid type is better then the other and no raid type is more complicated then the other when you actually know how to set them up and configure them. It depends on the work load, drive type, storage platform and budget. Raid1 maybe be fine in couple of T mom and pop shop on its a chumpy 4 port controller. But absolutely no real pro shop only does raid1 outside of that because its "safer". Yah, no.. Doesn't work that way.

Technically, raid 6 is safer then raid 1 with is dual fault failure tolerance..

Raid1 wouldnt have fixed ops original problem as it was already double faulted when he got it. Raid1 wouldn't have saved that. Backup volume missing, and then someone else over wrote one of the drives? They had bigger problems from day 1 then a raid issue.

I'm just saying Raid 1 is better if you have no real system management 24/24 at the top of the company which it seems was the case. I did that when I was often in charge to make the hardware and software work in company. In fact I wasn't in charge at all. I was in charge of completly something else, did that during my free time. It's just too easy to manage. You don't have to remember anything except the passwords. I switched from Linux to Windows for the file, scanner and print server, used the raid 1 from the chipset because there was nothing else and in fact I found I was happeir with it, and I even completely diminished to lower user rights all the desktop computers so nobody would be able to touch anything at the system level and installed software. And even desktops were on raid 1. All the problems and headaches were gone. This whole system ran for years without me touching at anything, excpet finding that somone pulled a plug or a HD to change (without losing any data or installing a system). No viruses, no troyans, no hacked software installed by people with temporary jobs, and most of it still runs now. It's simply no more time lost with computers at all and no need for an IT guy. But this goes well if your below around 20 computers I believe.

 

[Burticus]

Try this : R-Studio. Runs on Windows. I used it very long time ago, more 15 years ago, was called R-Tools, for a data recovery on a voluntarily linux raid 1 setup disassembled and boot erased by some guy. I restored all the data, but that data was still on it. Not sure how you configured your raid 5 setup and how the reaming disk was erased, maybe only half erased... that tool could try to read. In principle it tries to read, does not write anything. I also could read half of a dying hard drive. Software will try the best without hanging by inability of the OS to recognize the format of the drive.

https://www.r-studio.com/

If this $50 tool can do half of what it claims it can do, give it a shot. Worse case the OP is out $50. Send that system to a recovery house and be prepared to spend THOUSANDS and they guarantee nothing.

Ask me how I know this, LOL

Functional ackups are fundamental for business. And don't let bro dudes touch raid systems. Sorry man.

Worth $50 to try but I'm 90% certain you're hosed. 2 drive loss on raid 5 is bad. You might MAYBE get parts of it back but all of it is not gonna happen.

 

[Jandor]

If this $50 tool can do half of what it claims it can do, give it a shot. Worse case the OP is out $50. Send that system to a recovery house and be prepared to spend THOUSANDS and they guarantee nothing.

Ask me how I know this, LOL

Functional ackups are fundamental for business. And don't let bro dudes touch raid systems. Sorry man.

Worth $50 to try but I'm 90% certain you're hosed. 2 drive loss on raid 5 is bad. You might MAYBE get parts of it back but all of it is not gonna happen.

What I'm not so sure is that it erased data for real on the 3rd disk. A fast format doesn't. In that case if the software is smart enough, it may recreate most of the missing parts that were supposedly erased.

 

[drescherjm]

What I'm not so sure is that it erased data for real on the 3rd disk. A fast format doesn't. In that case if the software is smart enough, it may recreate most of the missing parts that were supposedly erased.

If linux was installed to the 3rd disk I would expect the first 10 or so GB of data would have been trashed after that it could depend on what filesystem was selected.

 

[cyclone3d]

150TB of data is not that sort of setup mentioned in the thread. So my rule applies in that situation. Raid 1 is safer, and much much easier to manage. Would this have been a raid 1 setup, it wouldn't have any such problem in case the raid setup might have been broken. In his case the problem occured at first because the raid setup has been broken and raid 5 recover is complicated. You really need to avoid any wrong move and also you cannot recreate easily your data by reading each disk (there are tools). All this is avoided by Raid 1 setups. They are a no-brainer.

UMMM.... RAID 5 is NOT complicated to "recover". If a drive fails, you replace the drive and it rebuilds automatically.

The problem is that the company that had the setup had incompetent people doing the IT work and they didn't replace the drive when it failed.
Then they brought in another random person who apparently should never have been allowed near a RAID 5 setup because they went and tried to install Linux on one of the remaining drives in the RAID setup which is a pretty sure way to make all remaining data completely useless.

And they also don't have any backups.

That company is pretty much screwed and it is their own fault.