Linux 10x3TB file system and RAID question

[ChineseStunna]

Hi all,

I'm currently in the process of upgrading my Linux MDADM RAID6 array and figured I ask some questions before proceeding and perhaps change configs while I have this chance.

Current Setup:
10x2TB running RAID6 through MDADM and using XFS on top on Debian (VM on ESXi), mostly just media/file storage, no application or VMs running on it.

Upgrading to:
10x3TB drives

Quesstions:
1) I'm considering a different FS than XFS, it has worked well but with larger and larger pools of data bit rot is starting to really concern me. I have my photography back up on this server with PAR data but would consider switching to ZFS or BTRFS for better data integrity protection. Thoughts?
2) If I'm just doing a direct migration, my current plan is to do a disk to disk dd of all the 2TBs to 3TBs a couple at a time. Once all the 3TBs have mirrored 2TBs, I will spin up the array and do XFS_grow on it to access the available space. Any problems/concerns with this plan and any thing I should watch out for?

Thanks!

 

[westrock2000]

Wouldn't the file system know that the drive itself is different? Even if the data itself is the same, will the software still see a different hardware ID and go into degraded mode? You might try just one first and see how it gets accepted back into the raid.

 

[westrock2000]

Based on what I read reading on Wikipedia, XFS isn't really that different than other Linux file systems. Where as ZFS is more specifically made to work with large amounts of data as well as keep data consistent. It has some really cool features, especially when it comes to checking data integrity.

 

[houkouonchi]

Hi all,

I'm currently in the process of upgrading my Linux MDADM RAID6 array and figured I ask some questions before proceeding and perhaps change configs while I have this chance.

Current Setup:
10x2TB running RAID6 through MDADM and using XFS on top on Debian (VM on ESXi), mostly just media/file storage, no application or VMs running on it.

Upgrading to:
10x3TB drives

Quesstions:
1) I'm considering a different FS than XFS, it has worked well but with larger and larger pools of data bit rot is starting to really concern me. I have my photography back up on this server with PAR data but would consider switching to ZFS or BTRFS for better data integrity protection. Thoughts?
2) If I'm just doing a direct migration, my current plan is to do a disk to disk dd of all the 2TBs to 3TBs a couple at a time. Once all the 3TBs have mirrored 2TBs, I will spin up the array and do XFS_grow on it to access the available space. Any problems/concerns with this plan and any thing I should watch out for?

Thanks!

1) I would just stick to XFS. BTRFS support is just not there yet and ZFS does not run natively on linux so I would personally avoid if on linux but that is just me. If you really want ZFS I would probably just run oracle solaris or something.

2). That should work if mdadm is stopped while you do it. Alternatively its running but the file-system is mounted read-only during cloning of the drives.

I actually didn't fully trust my areca and or seagate drives to last when I went from 20x1TB seagate to 20x2 TB hitachi so I basically mounted it in read-only mode and replaced each disk one by one letting the array rebuild each time (which is another option). I could have gone back to the old disks if I had a catastrophic failure. You should be prepared to have to re-create the mdadm metadata with the assume clean option though so be ready for that.

 

[ChineseStunna]

1) I would just stick to XFS. BTRFS support is just not there yet and ZFS does not run natively on linux so I would personally avoid if on linux but that is just me. If you really want ZFS I would probably just run oracle solaris or something.

2). That should work if mdadm is stopped while you do it. Alternatively its running but the file-system is mounted read-only during cloning of the drives.

I actually didn't fully trust my areca and or seagate drives to last when I went from 20x1TB seagate to 20x2 TB hitachi so I basically mounted it in read-only mode and replaced each disk one by one letting the array rebuild each time (which is another option). I could have gone back to the old disks if I had a catastrophic failure. You should be prepared to have to re-create the mdadm metadata with the assume clean option though so be ready for that.

Thanks! this is super helpful, I'm currently actually doing the array rebuild approach you mentioned. I think my current methods are a bit different though - I didn't mount the array at all, just started the array in mdadm with 9 out of 10 drives, "failed" the 1 2TB I took out, then "removed" it.
After that I "add" the 3TB that's put in it's place and the array it rebuilding as we speak. My concern with my current approach is that should there be catastrophic failure, metadata is screwed since the old drives are now resyncing with the 3TB so if it fails I don't think the old 2TB would just be directly plug and play without resyncing again (if that makes sense).

 

[brutalizer]

Lot of people run ZFS on Linux very successfully. Linux people says it is production-ready. Just google a bit.

 

[drescherjm]

Lot of people run ZFS on Linux very successfully. Linux people says it is production-ready. Just google a bit.

At work I have just started migrating from btrfs on top of mdadm raid6 to zfsonlinux raidz2 on several of my linux servers. I would do more conversions however I do not have the budget at the moment to be able to duplicate 15TB+ arrays (I do have tape backups but can't afford the downtime) so that I can't touch my larger arrays..

 

[dandragonrage]

1) I would just stick to XFS.

Not a good choice. XFS is an unsafe filesystem.

If you really want ZFS I would probably just run oracle solaris or something.

This advice is stupid. ZFSonLinux is excellent, with far fewer disadvantages for most of us than running Solaris would have. As someone who has run my server on multiple versions of OpenSolaris, Solaris 11/11, Solaris 11.1, and ZFSonLinux... ZoL is by far the best and by far the easiest to get good performance from (OpenSolaris, I found, often ends up limiting my SMB/NFS speeds to 30-40MB/s when Linux has no such problem even when Bonnie++ and NetIO benches on OS are fine). For production enterprise servers I would not consider it mature enough, but the only practical disadvantage is that they're slow to support new kernels (why hasn't 3.13 support been released? 3.13 pretty much just came out, but it's been in RC for long enough that ZoL really ought to be supporting it)

ZFS is the only decent option here, and BTRFS should still be considered alpha. Expect further disk format changes down the road (the other downside to software created by Linux users instead of Unix users).

 

[drescherjm]

only practical disadvantage is that they're slow to support new kernels (why hasn't 3.13 support been released? 3.13 pretty much just came out, but it's been in RC for long enough that ZoL really ought to be supporting it)

At home on the linux based htpc I tried to build spl for 3.13 only to have my os say the kernel was too new. I guess I will have to wait a bit. Probably best anyways since I do not like running kernels until at least a few patch levels..

 

[dandragonrage]

At home on the linux based htpc I tried to build spl for 3.13 only to have my os say the kernel was too new. I guess I will have to wait a bit. Probably best anyways since I do not like running kernels until at least a few patch levels..

What distro did you try that came with 3.13? Very few, if any, should be including it so far.

Personally I wouldn't recommend using a new kernel version until it's had a couple patches (e.g. 3.13.2 or something) anyway. Linux kernels that are x.y.0 should be considered beta - they always break significant amounts of things. Linux kernel "RCs" should be considered alphas.

 

[drescherjm]

What distro did you try that came with 3.13? Very few, if any, should be including it so far.

I use gentoo at home and at work both for over a decade. I tend to be more experimental at home than work. Although at work I do have servers with different levels of importance. I update the more important servers less often (usually after tested on several systems for weeks) than the ones that can survive some downtime. That is unless there is an important security fix.

Personally I wouldn't recommend using a new kernel version until it's had a couple patches (e.g. 3.13.2 or something) anyway. Linux kernels that are x.y.0 should be considered beta - they always break significant amounts of things. Linux kernel "RCs" should be considered alphas.

Agreed.

 

[Xamayon]

it has worked well but with larger and larger pools of data bit rot is starting to really concern me.

Since you are running raid 6, your array does have some bit rot resistance. As long as you run a consistency check frequently, have it correct any errors found, and don't run for long periods with a failed disk, your data is fairly safe. ZFS does have many benefits, and protects data from corruption caused by things other than bit rot, but if bit rot is your only concern, you don't necessarily need to abandon normal raid.

All file systems have their pluses and minuses, but XFS may not be the best choice for your critical data on its own. XFS is fast and awesome, but only concerns itself with the metadata. In the event of a system failure you can easily wind up with files that appear to be perfectly fine, but contain none of their actual data.

 

[lightp2]

10 x 2 TB RAID-6 approximately 16TB usable.
10 x 3 TB RAID-6 approximately 24TB usable.
multistage complication to get additional 8TB, hmm...

no distraction to your original plan, just another option
1. Buy 3 x 4TB in RAID-5 usable 8TB put in external enclosure

-- else if you have spare computer, and you really plan on doing this ----

1. 10x2TB RAID-6 usable 16TB on computer 1
2. 6 x 4TB RAID-6 usable 16TB on temporary computer 2, export file share. copy old data from computer 1 to computer 2.
3. When step 2 done. unmount 10x2TB RAID-6 on computer 1. Move 6 x 4TB RAID-6 from computer 2 to computer 1 if you really like to use computer 1. Mount 6x4TB RAID-6 on computer 1.
4. Since you have spare disks slots (original 10 disks, now only 6 active), you can then add 4 more 4TB disks to expand RAID-6 to upgrade capacity 10x4TB RAID-6 usable 32TB

why 6x4TB on computer 2 initially? Common computer only has 6-sata ports. If you can find more ports, obviously you can use 8x4TB or more.

In this case, the original SATA ports on computer 1 (current) also have some consideration. Are they SAS ports or through other special add-on controller? This scheme is possible if you use normal AHCI SATA Add-On Controller to complement the on-board sata ports. If you have very special hardware configuration, then it may not work.

Observation : original data disks are not modified. If the new RAID-6 set had issue, you can re-mount 10x2TB RAID-6 old disk set.