Best configuration for 12x4TB disks (ZFS)?

B

bleomycin

Limp Gawd
Joined
Aug 14, 2010
Messages
242
I currently have a 12x2TB drive raidz2 pool that is nearly full. I'm looking into rebuilding with 11-12x4TB drives. I'm not sure what the best compromise would be between speed and reliability.

Should I run:
2x6 drive raidz2 arrays
1x11 drive raidz3 (have to drop 1 drive for raidz3 for proper filesystem alignment with 4K drives, correct?)

something else?

Usable space and reliability are the main priorities, if i need to add some ssd's for ZIL/L2ARC for better performance i'm ok with that. I'm going to be running a few VM's on this but nothing super heavy, it's mostly bulk storage. I would like to achieve decent performance and i'm willing to pay a bit extra to achieve it.

Thanks!
 
Greetings

There is a discussion about that issue on this thread here, it is usually best to have the number of data drives as a power of 2 as the recordsize which has a default of 128KB (but can be other sizes) then divides cleanly into the number of drives you have e.g. in your case default 128KB / 8 data drives = 16KB per drive. Other non standard numbers of drives wastes more space. There is a website linked to in the first link I provided which is here. Kindly note that even if you follow this rule then 8 data drives does have slightly more overhead than expected compared to 4 or 16 data drives and this has something to do with ZFS internals in how it actually stores the data.

In your case for 12 drives I would just do a 10 drive Raid-Z2 (8 data + 2 parity) for the bulk storage as that only has the IOPS of a single disk but that does maximise storage space and probably a 2 way mirror with the remaining 2 drives for the VM's. If your VM's do a lot of disk writes then with ZFS being a COW (copy on write) filesystem then your data will get heavily fragmented so a ZIL/L2ARC is probably a good idea. If you have a lot of similar VM's you could also try dedupe for them but be aware of the large RAM overhead which I believe is around 330 bytes per recordsize.

Cheers
 
I recommend NAS4Free due to smooth interface and simple effective management front end.

As far as whatever array you want? Well that is debatable. With that many drives you are going to get a lot of performance out of doing a RZ2. I am running 8 2TB Western Digital RED drives and after doing an asinine amount of benchmarking etc... I was getting 550-600MBps reads and 450-500MBps write speed using motherboard SATA connectors. I am also running it on a Xeon E5-1620 (Have fun finding one of these badass chips, Trust me!) @ 3.6ghz stock clock. This is a socket 2011 and I have 3GB of RDIMM ECC on a SuperMicro Board with an Intel 10GB/s Short Reach Fiber to my Cisco Switch. So I do have quite overkill for my little 8 dive array.
 
Thank you both for the suggestion I appreciate it.

I've been running zfsonlinux for quite a while now and intend to stick with it. I supposed my biggest question/concern is with 4TB drives at what point is raidz2 insufficient? What is the maximum safe drive count per pool with 4TB drives in raidz2? The only reason I can see in my mind for running raidz2 over raidz3 in this situation is being able to split them into two 6 drive raidz2 pools to gain the extra performance and faster rebuild times? Either way I end up with the same amount of usable storage.

2x6 drive raidz2 arrays = 8 usable drives
11 drive raidz3 array = 8 usable drives

I'm beginning to lean towards the 2x6 drive raidz2 setup, but is raidz2 pushing it with even 6 drives at 4TB capacity?

Thanks!
 
One raidz3 would be safer than two raidz2, because if you loose three disks in raidz2 your data is toast. With raidz3 you must loose four disks before your data is toasted, which is lesser risk.

Here is a discussion with 16 disks:
http://hardforum.com/showthread.php?t=1779205

So, assuming you have 12 disks in a JBOD case, you could add another 4 disks in the PC case, to bring up the maximum to 16 if you wished.
 
Without RAID you need for all the drives to fail before you lose all of oyur data.

With RAID only a few drives failing will lose all of your data.

But data loss is not reliability.

To answer your question (speed/reliability): You want as little speed as you can tolerate. You want as much near line backup as you can afford.
 
Greetings

bleomycin said:
I supposed my biggest question/concern is with 4TB drives at what point is raidz2 insufficient?
Click to expand...

I suggest you read this then

http://queue.acm.org/detail.cfm?id=1670144

and also this

http://www.zdnet.com/blog/storage/why-raid-6-stops-working-in-2019/805

I think two parity drives is probably sufficient at this stage with Raid-Z3 being probably overkill, regardless of how robust your selection will be the probability of total failure of the arrays will always be greater than zero, hence backups are always necessary.

Cheers
 
Thanks everyone,

I do have complete offsite backups so that isn't my major concern. Avoiding downtime is my biggest concern, which is why I do want the most reliable raid configuration. Unfortunately 12 disks is the maximum, i don't have the space for more drives even in an additional external case. I'll have to do more research on the performance gains of 2x raidz2 vs 1xraidz3 and base my decision on that I suppose. Perhaps raidz3 with ssd's for zil/l2arc will be the best option in my case?
 
2x 6 raidz2 will offer vastly greater performance than a single large raidz3
 
GeorgeHR said:
But we don't know how much performance he needs.
Click to expand...

I don't have a specific performance requirement, this is a bulk storage/hobby VM server. With that said i'm always after as much performance as possible. Is it safe to say a 6x4TB raidz2 array is about as safe as my current 12x2TB raidz2 array?
 
bleomycin said:
I don't have a specific performance requirement, this is a bulk storage/hobby VM server. With that said i'm always after as much performance as possible. Is it safe to say a 6x4TB raidz2 array is about as safe as my current 12x2TB raidz2 array?
Click to expand...

Technically it would be safer, as in a 6 drive raidz2 you would have 33% devoted to parity,

whereas in a 12 drive raidz2 you would only have 16% devoted to parity.

safest of course would be a mirror configuration, with at least 50% devoted to parity (if not more in multiple mirror configs)

Size is only really a factor in rebuilding time, but if it came to that it would be at least equal safety.
 
I am a big fan of mirroring, too. It is just so much more resistant against major screwups. All you have to do is grab a single drive and run and you can read the data without doing anything fancy.

However, given a specific set of space requirements and financial power to buy drives a cleverly chosen higher raid can be the better option. Namely, how many drives can you have fail before you lose (first|last piece of) data?

Myself I partition my drives and then do different raid levels in different sections. This scheme also allows for a fraction of the resync time until the first such raid is fully synced up after a fail.
 
You must log in or register to reply here.
Back
Top