Creating a new ZFS RAID-Z system

I

ironicbadger

n00b
Joined
Nov 8, 2013
Messages
28
Hi guys,

Looking for some advice on the optimal config for my home server, which I've decided to migrate from unRAID to ZFS due to performance and bitrot prevention. I have checked out and discounted snapraid and aufs as I want realtime parity computation, plus the performance figures I've seen posted online for ZFS are a real draw along with CoW AND snapshots! Another goal is learning for me, ready for some interviews I will be lining up later in the year.

Hardware

I will be running this with a Xeon 1245 v3, Supermicro X10SAT, 6 (or 7 depending on the advice following this post) 3TB drives, a Supermicro SAS-AOC-SASLP-MV8 card and 32GB RAM (non ECC as I'm a poor student!).

Software

For better or for worse I want this box to do a lot of stuff at once. I'm a big fan of virtualisation and will rely on it heavily to achieve everything I want to do. This one box will be my fileserver, my main XBMC instance and also my desktop PC in one.

The desktop portion will be probably a Windows VM with a GPU passed through via vt-d. The main XBMC instance will use the CPU graphics from the host OS, with ZFS running along side on the host.

I will be using Arch Linux as the host OS. That is not up for discussion in my mind! :cool:

What I really wanted advice on was the actual setup of the drives into pools.

RAID-Z2

I have 7 3TB drives in my possession right now, I want to be able to withstand a double failure and want an 'optimal' configuration. It is worth saying that my house is only wired to gigabit ethernet, though a few of the VMs I'll be running will have higher than that access via the bridge NIC provided by Xen or KVM. So my goal is to have a write speed in excess of 400MB/s, and read speeds hopefully even greater than that.

I have read in quite a few places, this being one, that for a double parity situation I want to use 6 drives in a RAID-Z2 configuration. I could I suppose add my 7th drive to this config, but what sort of performance penalty will I incur? Can I just have a 7 drive RAID-Z2 config?

Using this RAID calculator I've calculated with Z2 for 6 drives = 10.9TB space, 7 drives = 13.6TB usable space. The latter is a nice balance if you ask me between size / cost / reliability but what I can't answer is the performance hit.

Command to actually create the array

I've been playing around with a few of the drives (only 3 are clear of data right, I"m copying stuff around right now ready to migrate in a couple of days) and wanted to clear up the syntax for creating the actual array ahead of time.

Does this look correct? Source

Code: 
zpool create myraidz2 raidz2 /dev/disk1 /dev/disk2 /dev/disk3 /dev/disk4 /dev/disk5 /dev/disk6 /dev/disk7

Monitoring and management

One of the things I like most about unRAID is the crappy webGUI which gives me a quick overview of what's going quickly. With ZFS-on-Linux what tools are available to me which give a similar output to the

Code: 
zpool status myraidz2

If you made it this far, I salute you. Thanks for your input in advance!
 
I would recommend running omnios or something like that as a storage appliance, passing an HBA or something to it using vt-d, then you can you napp-it for management.
 
Right, that approach isn't too far removed from what I'm doing now with the storage on a VM (with unRAID).

One of the main reasons I'm looking at doing this is to get the storage natively on the host.
 
What does that gain you though? FWIW, I am still not convinced ZoL is quite ready for prime time. With vt-d passthrough, performance is near bare-metal...
 
ironicbadger said:
Does this look correct? Source

Code: 
zpool create myraidz2 raidz2 /dev/disk1 /dev/disk2 /dev/disk3 /dev/disk4 /dev/disk5 /dev/disk6 /dev/disk7
Click to expand...

Small bit of advice here about something that bit me rather severely once.
use /dev/disk/by-id/ instead of the names such as /dev/sdb1, etc.. That way if the drives should ever move (like they did once on me after mucking about in the case) the array will still work. It's simple enough to fix if it does happen but better to avoid in the first place.

danswartz said:
What does that gain you though? FWIW, I am still not convinced ZoL is quite ready for prime time. With vt-d passthrough, performance is near bare-metal...
Click to expand...
That old "argument" comes up again. ZoL has been "ready for prime time" for a long time now.
 
FnordMan said:
Small bit of advice here about something that bit me rather severely once.
use /dev/disk/by-id/ instead of the names such as /dev/sdb1, etc.. That way if the drives should ever move (like they did once on me after mucking about in the case) the array will still work. It's simple enough to fix if it does happen but better to avoid in the first place.


That old "argument" comes up again. ZoL has been "ready for prime time" for a long time now.
Click to expand...

Thanks for the by-id tip, It's that kind of thing I really wanted!

ZoL argument in my case doesn't matter anyway because I'm a home user with data that I can afford to loose. Anything worth a damn is replicated more times than I can count via BTsync and friends houses.
 
My friend and I have been using ZoL for a couple months now, version 0.6.2. 0.6.0 is where it really became pretty stable and some people started calling it "production ready".

Or at least Lawrence Livermore National Laboratory uses ZoL in production with over 1PB of data on it now too.

0.6.3 should be out soon. Development is coming along pretty fast and now that the OpenZFS project exists they are all working together to get all the downstream stuff from Illumos into BSD and Linux fast and reliably while also documenting and working through the differences in the platforms.

My friend and my experience with it over the past few months has been flawless at least. I've gone through some hardware issues including a bad SATA cable, bad power cable (so dropped and erroring disks) and a completely failed disk and ZoL has handled it all perfectly and pretty automatically so far.
 
"That old "argument" comes up again. ZoL has been "ready for prime time" for a long time now. "

Well, look, I'm no hater. And you don't help your credibility with snarky quote marks like that. Do you subscribe to the ZoL list? I do, and I stopped using it because of some fundamental issues that have not been resolved yet. In fairness to Brian and his crew, they are not things directly under their purview. Basic serving up of data works very reliably, yes. I'm talking about things like brittle interactions between ZoL and udev (so that devices don't show up at the right time, zvols are missing on boot, etc...) There are other edge cases I can't remember offhand, but perusal of the mailing list will turn them up. I'm no huge solaris fan, but being able to install something like omnios and have everything just work with my LSI HBA is nice. With ZoL, I had to disable autostart of the iscsi target daemon, forcibly remount and share some datasets, then manually start ietd (in /etc/rc.local) to get around some of these issues. Not being able to have a zfs root pool that is officially supported was another strike against it. Again, that is no knock on Brian and company, just the way it is. Maybe instead of 'not ready for prime time', I should have said something like 'not GA yet...', and it surely isn't. The OP has said he's fine with losing data because he can get it back. That of course, is his call. I use ZFS for a home setup with multiple production VMs for things like email, web, internet access, etc... My wife uses a win7 VM to telecommute, so my tolerance for bugs and glitches is lower than the OP's.
 
I use ZoL as my root for two arch linux installs perfectly. I've been running both for a few months now and have had no issues. One is a 6 drive Z2 and the other is dual SSD in mirror.

I haven't had drive failures yet so I can't personally verify how well it handles that but no issues otherwise.
 
Yeah I think ZoL can be good depending on your use case. I don't use it on root and I don't use zvols. I didn't have to do anything special to make it work it just works like I would expect "out of the box".

I can see why it's not "ready" for everyone, but it certainly I think can be ready and stable for some.
 
Agreed. For the basic use case of serving up storage, it works fine. Some HBAs present disks more slowly then expected, and given when the ZFS driver loads, that can be a problem. It's easy to blame the HBA, but this all 'just works' on FreeBSD or any flavor of solaris I've used, due to their different device architecture. I'm sure this will all shake out eventually, but for now, not having to band-aid things in rc.local to get all the services to come up properly, is worth it to me.
 
ironicbadger said:
I will be running this with a Xeon 1245 v3, Supermicro X10SAT, 6 (or 7 depending on the advice following this post) 3TB drives, a Supermicro SAS-AOC-SASLP-MV8 card and 32GB RAM (non ECC as I'm a poor student!).
Click to expand...

That post died right there. You can afford all this server-class hardware and then cheap out on the most important part?
 
With 7 drives I would run 2x raidz1 vdevs with a hot spare.

Twice the write performance than the 6 drive raidz2.
 
TCM2 said:
That post died right there. You can afford all this server-class hardware and then cheap out on the most important part?
Click to expand...

That board is more of a workstation board. No IPMI, no onboard Type-A USB, has audio, etc. That being said i'd go ECC with the Xeon and a board with IPMI so i can run headless easier.
 
bds1904 said:
With 7 drives I would run 2x raidz1 vdevs with a hot spare.
Twice the write performance than the 6 drive raidz2.
Click to expand...
You mean IOPs.

@OP: I'm a happy user running about 25 production VMs on top of KVM + ZoL (I'm not using ZVOLs though).
 
I also found the memory part strange unless he's already got it, ECC RAM is not much more expensive than non ECC one, especially since memory prices have rebounded.

bds1904 said:
With 7 drives I would run 2x raidz1 vdevs with a hot spare.

Twice the write performance than the 6 drive raidz2.
Click to expand...

The OP makes a mistake and you make the opposite mistake. He only talks about sequential speed, you only talk about IOPS.

He might need both if he plans to put the VMs on the HDDs, then I would suggest a SSD or two, but otherwise, a single vdev can get him the 400MB/s sequential he asked for.
 
I am now a ZFS junkie.

Screen%20Shot%202014-01-24%20at%2014.19.47.png


I already have the RAM btw, bought it a few years ago hence why I'm not upgrading it! I wish for ECC too.

I am considering going for 2 3x vDevs as well, but can't decide this figure above is from a single 7 drive pool and that really is fast enough for me as I have my VMs running from an SSD separated already.
 
Aesma said:
The OP makes a mistake and you make the opposite mistake. He only talks about sequential speed, you only talk about IOPS.

He might need both if he plans to put the VMs on the HDDs, then I would suggest a SSD or two, but otherwise, a single vdev can get him the 400MB/s sequential he asked for.
Click to expand...

Everything I've ever seen and read (including over on the virtualization board) says that for VMs, IOPs is more important than sequential throughput. Just because the OP mentioned wanting to shoot for a certain seq rate, doesn't make it the proper target.
 
You must log in or register to reply here.
Back
Top