Increasing server capacity intelligently - adding vdev mirrors vs. replacing disks

iroc409

[H]ard|Gawd
Joined
Jun 17, 2006
Messages
1,385
My storage array just hit 80% capacity on my FreeBSD/ZFS file server, so I guess I need to start thinking about increasing capacity. I want to plan it appropriately to make sure I'm not chasing dead ends or end up with a whole pile of parts I don't need from projects I didn't do like I have now. The server is an older Dell T110ii and the array is 4x4TB WD Red in mirrored pairs. I think I got the server and the drives at about the same time, though not positive, but the drives are dated 2014.

I like the server, though it's getting long in the tooth it's well built, quiet, and has been very reliable--and I enjoy that it's a prebuilt. The only real problem I've had with it is needing an add-on USB 3 card and having one fail on me already (but it was pretty old). The chassis is limited with 4x3.5" drives and 2x5" bays (one of which has an Icy Dock 4x2.5" setup). However, I suppose it is getting a little old. I'd like to build a new server, but don't cherish the thought of spending the money and no need to replace functioning gear for home use I suppose. I am open to any and all suggestions, but have kind of whittled it down to these three main options:

1. Swap the existing 4TB drives for larger ones in the existing T110ii, replacing existing drives and doing an online expansion. Pros: quick & easy, cons: cost of new larger hard drives, age of hardware
2. I have a Supermicro X9SCL-F collecting dust, along with almost every other part I'd need to swap over to a different system, and add mirror vdevs to the existing array. I actually already have two more 4TB Red drives, one from 2014 and one 2018. The only thing I would need to pick up is a chassis. Cons: age of hardware, increased electrical use (?), pros: cheapest and painless way to do it
3. Build a new server and transfer files over. Would it be ridiculous to build just a mirror of a pair of 12TB drives? It would probably keep me in line for a couple of years, by then bigger hard drives will be around. I like the idea of streamlining my network gear to a smaller physical footprint. Pros: new gear, new longevity, cons: significant cost

Just looking for some input so I make sure and plan this out a little better this time.
 
Option 2 seems the most reasonable.

That's the direction I'm leaning as it's the cheapest.

Anyone have solid case recommendations? I found a Chenbro that looks pretty reasonable but it is a little pricey. Could probably use it for years though.
 
I've built my NAS servers in Fractal Design Define R5 and XL R2, depending on storage needs. The XL is good for 12+ drives.
 
The R5 looks perfect, and quite a bit cheaper and smaller than the Chenbro in the end. That should work fine!

I'm considering now if I should leave the array in mirrors, or destroy the mirrored pairs and convert to raidz2. With 6 disks that would seem much more efficient, or I could go to a full 8x4TB and have more space than I'll use in years. I'd have to restore from backups, which I don't cherish, but it would probably be worthwhile. My backups are single-drive ZFS volumes on external USB.
 
The R5 looks perfect, and quite a bit cheaper and smaller than the Chenbro in the end. That should work fine!

I'm considering now if I should leave the array in mirrors, or destroy the mirrored pairs and convert to raidz2. With 6 disks that would seem much more efficient, or I could go to a full 8x4TB and have more space than I'll use in years. I'd have to restore from backups, which I don't cherish, but it would probably be worthwhile. My backups are single-drive ZFS volumes on external USB.
I went with mirrors because of the much easier and cheaper capacity upgrade options. You would have to update the raidz2 pool in batches of six, while mirrors only require 2 at a time.
 
I went with mirrors because of the much easier and cheaper capacity upgrade options. You would have to update the raidz2 pool in batches of six, while mirrors only require 2 at a time.


That makes complete sense, and I might still stick with it for that reason (or that I don't have to restore from backup). My only real thought there is that I get more efficient storage with arguably better protection converting to raidz2. I have 5.87TB of data currently, and it's taken me years to develop that. Going to 12TB, I can see maybe in the next couple years filling that up and having to upgrade. Going straight to 16 or 24TB with 6x4/8x4, I'll probably be well past the hardware's useful life (or failure) such that it would probably require a complete replacement. I don't torrent/download or horde any data, I make digital copies of the CD/DVDs I own (the bulk of which is done) and my photography. The photography is building data pretty fast, but I've been a lot better about culling my bad stuff.

On the other hand, it's always nice to retain the ability to just upgrade mirrors and the like. Even past hardware life, as you say I can just upgrade in batches which is easier to stomach.
 
Yeah, RAIDZ2 would provide more storage efficiency, which is a big plus. Protection is better and worse at the same time. You do get any two failed drives resiliency against RAID10's two from the same mirror pool loss, but at the same time up to half of all drives could fail without data loss, and rebuilds are almost instant compared to Z2.

It was hard for me to choose, but relatively affordable HDD prices per GB to offset capacity loss and easier upgrade options steered me towards RAID10 in the end.
 
The R5 came in today. It's so much larger than my T110; I'm going to have to re-arrange my network shelf height. Nicely made though, I only have experience with one other Fractal case and I am happy with both.

I think I've decided the best way to go for now is to rebuild the new server, and just add the two new drives in mirrors with the additional capacity. Once it's all stable, I'll decide if I need to change it or just leave the array as-is. I need to update some of my backups and stuff so that buys some time.
 
The R5 came in today. It's so much larger than my T110; I'm going to have to re-arrange my network shelf height. Nicely made though, I only have experience with one other Fractal case and I am happy with both.

I think I've decided the best way to go for now is to rebuild the new server, and just add the two new drives in mirrors with the additional capacity. Once it's all stable, I'll decide if I need to change it or just leave the array as-is. I need to update some of my backups and stuff so that buys some time.

I have the R5 (new gaming rig.) Love it...
 
BTW, there is a pull request for a method to add drives to raidz without needing to implement block pointer rewrite. From past experience (with TRIM, encryption and other features) I expect this feature to be merged sometime in the next 2 years.

https://github.com/zfsonlinux/zfs/pull/8853
There is a video about the theory here:
 
Last edited:
  • Like
Reactions: Meeho
like this
Back
Top