Need NAS software RAID and filesystem suggestion

E

ezno_matrix

Limp Gawd
Joined
Feb 9, 2012
Messages
134
Now that my NAS is the most important device in my household, I am looking for suggestion to reduce my downtime as I outgrow the current disks. Firstly, my data is important but only a small amount is irreplaceable. The irreplaceable stuff is backed up in multiple location. That being said, I still would not want to go through the hassle of restoring a single bit of it.

Requirements:
1. Preferable runs on Linux. But I can deal with any of the *NIX so no issue switching if one of these will work. I have no experience with Windows beyond an end user desktop and other than email and web, most of time is spent remoting to a *NIX machine. Not sure if OSX is an option.
2. Full disk encryption must be supported.
3. I need maximum storage of no less than n-4 disks (as I get today with two RAID6 volumes).
4. Should be considered very stable and reliable.
5. FREE (as in speech) software preferred but I am open if the closed software features are compelling.

Here is the challenge;
I have a 13x1.5 disk RAID6 array that is now showing its age. The disks are starting to throw prefailure SMART errors and needs to be replaced. I cannot replace all the disks at once and I want to replace the failing disk(s) with 3TB drive. As the disks fails I would like to:
1. Remove failed disk from the array
2. Shrink the filesystem and RAID volume to the size of n-1 disk. This should take no more than a day if the filesystem has to be unmounted. (I tried to shrink a 15TB ext4 filesystem that was 90% full. It ran for a week but never completed)
3. Add a new 3TB drive and start a new volume. The volume with 3TB disks will go from JBOD to RAID1 to RAID5 to RAID6 as I replace more 1.5TB disks. This needs to be doable with a mounted filesystem.

Is this safely doable? Am I going at this in a completely wrong way?

My current solution is a very risky disk juggling where I fail disks from another RAID6 volumes, copy the data then rebuild the 1.5TB RAID with n-1 disk. With data copy and RAID rebuilds, it takes about a week to complete the entire operation which is no longer acceptable.
 
Last edited:
"I need maximum storage of no less than n-4 disks (as I get today with two RAID volumes)."

What do you do right now, exactly?
 
I had 13x1.5TB disks setup in RAID6 with about 15TB of data. I had another RAID6 with 10x3TB disks with about 5TB free space. This is in a 24 bay chassis with my OS drive in the other bay. The OS takes up about 2GB and takes less than 10min to restore, so no redundancy needed for the OS drive. Here is what I did when a 1.5TB drive fails:
1. Purchased a new 3TB drive.
2. Removed the failed disk from the 13x1.5TB disks array leaving it in a degraded state with 12x1.5TB disks.
3. Swapped the failed 1.5TB disk for the new 3TB disk.
4. Added the new 3TB drive to 10x3TB disks RAID6 array and grew that array to 11x3TB disks with 8TB free space.
5. Took the NAS offline.
6. Copied 8TB of data from the degraded 12x1.5TB disks array to the 11x3TB disks array. This left 7TB of data on the 12x1.5TB disks array.
7. Failed and removed two disks from the 11x3TB disks array. Stopped the 11x3TB disks array (now running in degraded state with 9x3TB disks).
8. Copied 6TB of data from the degraded 12x1.5TB disks array to the two 3TB disk removed from the 11x3TB array. This left 1TB on 12x1.5TB array.
9. Failed and removed one disk from the 12x1.5TB disks array (now further degraded to 11x1.5TB disks).
10. Copied the remaining 1TB data from the 11x1.5TB disks array to the 1.5TB disk that was removed.
11. Created a new RAID6 array with 11x1.5TB disks.
12. Copied the data from the 2x3TB and the 1.5TB disks to the new 11x1.5TB array.
13. Started the degraded 9x3TB disks array and added the two 3TB disk that were removed. Rebuild array to the redundant 11x3TB state.
14. Brought the NAS online.
15. Added the 1.5TB disk to the 11x1.5TB disks array and grew the array to 12x1.5TB disk.
16. Redistributed data between the two arrays to free up space on the 11x3TB disks array.

This process took about a week, with the NAS being offline for most of that week (the first 5 steps took about a day). This downtime is no longer acceptable and I need a faster method that will allow me to shrink the filesystem and reduce the RAID array to n-1 disks.
 
FWIW, I do not create RAIDs (in Linux md) with more than 1 TB per-disk slice. That makes them much more nimble, although of course that limits your filesystem size to (n-2)TB and you will have to symlink around when individual directories grow too big.

When I replace smaller disks with bigger disks I create new arrays in the remainder of the new disks, I do not grow the arrays.

There is some pain with having to move stuff between filesystems sometimes to get around space limitations but it's not too bad and doesn't take much longer than your full copies do now.
 
So the larger the disk, the more RAID it belongs? How exactly do you slice the disks?
 
bexamous said:
why do that?
Click to expand...

To keep filesystems at manageable sizes. Just for starters when a disk dies I will resync the most important filesystem first and it's all set and secure in a fraction of the time than the whole disk takes.

I also use different filesystem types and sometimes different RAID levels. And I want the option of quickly draining an entire filesystem, where quickly means at most 1 TB / disk.

These 100+ hour procedures that we discuss here are a problem if you have very large arrays on 7200 rpm regular drives.
 
Wow this juggling is crazy, especially since you don't mention backups !

If the OS is only 2GB why not put that drive on the NAS motherboard or something, maybe even on a USB stick ?
 
uOpt, I have been thinking about your solution and I have two concerns:
1. It is not very scalable to keep 1TB partitions (what happens when I got to 6TB drives?)
2. In my case, it would be a lot of filesystems and partitions to manage. Though, I would throw LVM on top to simplify things.

However, you did point me in the right direction. Instead of the 1TB limit, I would keep the partitions to the size of the smallest disk in server and keep growing my RAID to include the partitions. I.E., for the failed 1.5TB disk, I would add a new 3TB disk with 2x1.5TB partitions. One of the two partitions would be added to the 12x1.5TB disks degraded RAID6 array and I would start a new RAID with the remaining 1.5TB partition. As more 1.5TB disks fails, I would keep adding 3TB disks with 2 partitions and keep growing both arrays. All this can be done with the server online.

I am still left with the challenge of switching to smaller number of larger disks, such as switching the 12x1.5TB disks RAID6 to a 5x4TB disks RAID5.

Before I attempt your suggestion, I am currently playing around with btrfs and I am really warming up to this filesystem. I always considered it lacking in maturity for my servers, but it is working fine on a few test disks. Btrfs shrinking was very fast on a almost full 3TB disk. Unfortunately, I do not have much spear disks to test. I think I will bite the bullet and throw it on the RAID that has mostly static data. I could always roll back to ext4 if I still face the shrinking problem.
 
Aesma said:
Wow this juggling is crazy, especially since you don't mention backups !
Click to expand...
From the first post in the thread:
Firstly, my data is important but only a small amount is irreplaceable. The irreplaceable stuff is backed up in multiple location.
Click to expand...

Aesma said:
If the OS is only 2GB why not put that drive on the NAS motherboard or something, maybe even on a USB stick ?
Click to expand...

I considered that, but I have a Gigabyte board that does weird things when booting from a USB. I think it is a boot sector size issue. I have also been lazy and haven't switched to UUIDs so when I add disk, the USB disk gets shifted and fails to boot. This always crops up when I'm out of town and need to do a remote reboot. I will most likely switch to network boot when I need the bay that contains the OS disk.
 
Aesma said:
Wow this juggling is crazy, especially since you don't mention backups !

If the OS is only 2GB why not put that drive on the NAS motherboard or something, maybe even on a USB stick ?
Click to expand...

The root filesystem is very high redundancy. With 5 disks I split it into 1 3-disk and 1 2-disk arrays that gets synced over when I "like" the OS.

ezno_matrix said:
uOpt, I have been thinking about your solution and I have two concerns:
1. It is not very scalable to keep 1TB partitions (what happens when I got to 6TB drives?)
2. In my case, it would be a lot of filesystems and partitions to manage. Though, I would throw LVM on top to simplify things.

However, you did point me in the right direction. Instead of the 1TB limit, I would keep the partitions to the size of the smallest disk in server and keep growing my RAID to include the partitions. I.E., for the failed 1.5TB disk, I would add a new 3TB disk with 2x1.5TB partitions. One of the two partitions would be added to the 12x1.5TB disks degraded RAID6 array and I would start a new RAID with the remaining 1.5TB partition. As more 1.5TB disks fails, I would keep adding 3TB disks with 2 partitions and keep growing both arrays. All this can be done with the server online.

I am still left with the challenge of switching to smaller number of larger disks, such as switching the 12x1.5TB disks RAID6 to a 5x4TB disks RAID5.

Before I attempt your suggestion, I am currently playing around with btrfs and I am really warming up to this filesystem. I always considered it lacking in maturity for my servers, but it is working fine on a few test disks. Btrfs shrinking was very fast on a almost full 3TB disk. Unfortunately, I do not have much spear disks to test. I think I will bite the bullet and throw it on the RAID that has mostly static data. I could always roll back to ext4 if I still face the shrinking problem.
Click to expand...

I hate LVM. It's true, however, that I was about to go instead of single disks under the raids to always have a mirror (raid1) with 1 disk as the base for every array, so that I can replace disks later.

Managing my zoo isn't too bad since I have several very diverse data sets, each not exceeding the capacity of the filesystems, a filesystem being 3 TB net.
 
You must log in or register to reply here.
Back
Top