• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

ZFS vdev setup

D

dev_null

n00b
Joined
Sep 30, 2010
Messages
10
Hello!
I am building a ZFS server for storage. It will consist of 12 WD red 3tb hard drives. At first I thought of having 6 mirrors, then I thought what if a hdd dies in one of the mirrors, but the other has bad sectors? Ive lost data (chime in if I am incorrect here). Then I thought about 3 4 hdd raidz arrays - what if something similar happens and 2 drives have issues in one of the vdevs? So I think I have decided on this:

2 vdevs of 6 3tb drives in raid z2 each. I plan on testing this once the rest of my drives come in. I wanted to see if any of you have a similar setup and if you could comment on 1) performace - will this be able to populate gigabit read/write in full duplex with no issues? honestly that is all that matters to me. what kind of speeds should I expect? 2) each vdev will be approx 18 tb total including parity. what kind of resilvering times should I expect for vdevs this size if i have to replace a drive?

If there is a better setup that i am not thinking of, feel free to chime in as well - ive read through some of the posts on this forum. I thought about a giant raidz3, but if I need to expand later, I am pretty sure Id rather add 6 hdds at once then 12. I do not plan on using a hot spare.

thanks!
-dev_null
 
Last edited:
For 12 disks and home or soho use, raidz2 is the best ...

1.) Run tests, for example latest Bonnie++, then it is up you ...

Supermicro X9SCM-F
Intel Xeon E3-1230
2x AOC-USAS2-L8E
16 GB ECC RAM
Solaris 11.1 latest
ZFS pool 2x6 raidz2

Seq read
Seq write

You can notice, how is the dataset performance affected by ZFS options such as atime, compression and encryption during sintetic benchmark.

Kind regards,
Ivan
 
Last edited:
For me, its definitely mirrors all the way. I figure you get the performance as well as redundancy.

I understand that if the second drive in the mirror is goosed then you have a problem but I'm hoping that regular scrubbing will avoid this.

If my logic is warped, someone please tell me as I'm just about to put a box based on this in the datacenter.
 
I go with raid10. I'm not particularly worried about two drives dying in the same mirror, or one dying and the other having bad sectors. I use SAS nearline drives, and scrub weekly. raid10 gets you 2X the read IOPs of your spindles (assuming random I/O).
 
In my opinion, the best would be firstly answer the simple question.

What is the main purpose of the array ?

Then you will be able to find out, what type of RAID do you need regarding performance and also how important are your data.

For example I don`t need I/O, it is for home use, garbage collection of useles data with minimal value ...
 
I'm doing something similar with 16 drives and my plan is a single RAIDZ3 vdev. Performance is good enough for pure storage and 3 drives can die on me. I would never trust mirrors without at least a double backup, all in all a very expensive endeavor since the mirrors already rob me of half the capacity.

Now if we're talking database or virtual machines storage, that's another story.
 
ESXi storage: I highly recommend raid10. I know you lose 1/2 your storage but IOPs is important for a datastore...
 
I think I'm covered Dan, I've got 8 x 1TB Barracudas in 4 mirrored vdevs and then I've added 2x256GB Samsung 840 Pro SSDs as L2ARC. They are striped, I think one should be well sufficient but I had a spare!

Its for 4 hypervisors, the storage box has two dual port 4GB FC cards in and there is a 4GB FC card in each HV.

Storage box board has an i3 processor and 32GB ECC RAM
 
ESXi datastores - with spindles, use mirrors all the way -
I went to SSD only pools and Raid-Z2

Price and reliability of SSD are now more than good enough and they
lift I/O from several hundreds to several thousands/s
 
I too wanted an all SSD pool but had a bad experience with some cheaper Crucial drives.

These are going in the datacenter hence the spinner/l2arc combo.

I don't need much in the way of random iops and though I know caching is in play I'm showing about 20K random on my setup.

I've got an iscsi feed from the datacenter's clustered san that gives me 50GB for my crucial stuff.

I've been closely following the progress of the OI+Napp-it clustering thread.
 
1 x 12 disk Z3

2 x 6 disk Z2

6 mirrors

all are acceptable but you didn't mention anything about performance requirements so nobody can really answer you properly.

if this is just a home box for storing media, i would do 12 disk z3. you lose the least space and have the best throughput that way. random IO is limited to the IOPs of a single disk but you don't need random IO for media streaming.
 
IMO, the three sensible options are:

Six mirrors for maximum performance, at the expense of storage capacity. Drive failure protection isn't the best, but if you have decent backups/failovers or replacement disks on hand then you may choose to accept that risk.
This is equivalent to a RAID-10.

Two 6-drive RAID-Z2s for a decent compromise. Effectively this would have the I/O performance of two drives in RAID-0. Best protection against drive failure of the three options, and not too much capacity wasted.
It's like a RAID-60.

One 12-drive RAID-Z2 for maximum capacity at the expense of performance. Effectively this would have the I/O performance of a single drive. Protection against drive failure is decent but not quite as good as splitting it into two RAID-Z2s.

Hope that makes sense.
 
_Gea said:
I went to SSD only pools and Raid-Z2
Click to expand...

What SSD's are you using for ESXi datastores? Enterprise drives, or cheaper consumer drives? Curious what your experience has been like with a specific model.
 
Oops, I thought I covered all bases. This is not just a media server, but will store ALL my stuff (school, taxes, everything important as well as non-important), so I'm interested in an appropriate balance of performance and protection. My performance requirements are simply to fill gigabit full duplex. I thought about giant z3, but I also thought about upgradability. 6 drives is easier and cheaper than 12 and I would expect far easier to resilver. Can anyone comment on resilvering times of a 6 drive vdev?
 
dsumike said:
What SSD's are you using for ESXi datastores? Enterprise drives, or cheaper consumer drives? Curious what your experience has been like with a specific model.
Click to expand...

Mostly I use Intel 320 and Winkom Pro (SATA III, Sandforce SF 2281 syncronous Intel MLC Chips with supercap).

Failure rate: One of about 50 the last 6 months but thats too less values to say 4% per year but I suppose its not worser.

What I do
- limit usage to max 3-5 years
- try to stay below 60% fillrate
- use a dedicated ZIL to avoid massively small writes (ZeusRAM, SLC SSD)
 
dev_null said:
Oops, I thought I covered all bases. This is not just a media server, but will store ALL my stuff (school, taxes, everything important as well as non-important), so I'm interested in an appropriate balance of performance and protection. My performance requirements are simply to fill gigabit full duplex. I thought about giant z3, but I also thought about upgradability. 6 drives is easier and cheaper than 12 and I would expect far easier to resilver. Can anyone comment on resilvering times of a 6 drive vdev?
Click to expand...

Resilver time depend on size, speed and usage because it is a background process with low priority.

On a raid-Z2 vdev of six faster 2TB disks I would expect a resilver "overnight".
If you expect growing capacity needs (we all do), I would also use 2 x vdev, best 2 x Raid Z2 so you can update half of the pool with larger disks.
 
dev_null said:
Oops, I thought I covered all bases. This is not just a media server, but will store ALL my stuff (school, taxes, everything important as well as non-important), so I'm interested in an appropriate balance of performance and protection. My performance requirements are simply to fill gigabit full duplex. I thought about giant z3, but I also thought about upgradability. 6 drives is easier and cheaper than 12 and I would expect far easier to resilver. Can anyone comment on resilvering times of a 6 drive vdev?
Click to expand...

Sounds like this is just a personal server - single user, maybe a few? In that case ignore the performance considerations. Any configuration will be plenty fast enough to serve your media and documents. If this were a SAN for 1000 users then it might be worth worrying about, but you didn't say :p

Don't worry too much about the resilver times. You're talking about a matter or hours or maybe days - not weeks. The chances of a second drive failing within that time-frame are very low, and if it does happen you should have a backup available to restore from. If you're still worried, how about keeping a coldspare?
 
You must log in or register to reply here.
Back
Top