ZFS Education Needed

[coolrunnings]

A bit of background: I have an HP ML110 G7 server. For those unfamiliar with the series, they have 4 SATA/SAS hotswap bays. I currently have 4x 1tb WD Black Edition 7200rpm SATA 3G drives in it. I am getting an IBM M1015 card so I can run ESXi and do an all-in-one server. I plan to run 3-4 VMs at a time off of this as well as stream large data files (backup images) to it frequently. I am wondering about the best way to set up the pool for both expansion as well as performance. I currently have 3 drives set up in a RAIDZ and with this can almost saturate gigabit ethernet. On to my questions:

1. Can I set up the drives in pairs of mirrors, then add them all to one pool to combine them, I am thinking this would allow me to add another pair of 1tb drives in the future for a total of 6 drives. Would this give me the same level of redundancy as a RAIDZ3 pool but better speed or am I misunderstanding how this works? Basically I'm trying to get around the destroy/rebuild the pool problem when adding storage but I don't want to sacrifice redundancy or performance.

2. I have a second server - a Dell PowerEdge T110 (g1, not g2). Currently this has 3x 2tb Samsung F4 drives in it (yep, 4k drives). I am thinking of making this into a second all-in-one with a RAIDZ1 for the sole purpose of backing up large data files and backups of the first all-in-one server. Will the 4k cluster size cause me performance/reliability problems doing this? Would it be better to get another F4 drive and do 2 mirrors instead?

Thanks for the help guys! Oh, and _GEA - YOU ROCK!!! Thanks for all your hard work on Napp-IT. I'm in the process of testing this and will see if I can get you some commercial accounts that will donate to the project.

 

[danswartz]

"Can I set up the drives in pairs of mirrors, then add them all to one pool to combine them"

Not sure I'm parsing this right - it sounds like you want the zfs equivalent of raid10? If so, yes, this works just fine - I have 3 2-way mirrors of WD blue 640GB drives striped that way.

 

[madrebel]

mirrors won't give you the availability of triple parity.

 

[coolrunnings]

So basically I should do a RAIDZ2 volume with all 4 drives if I want the most data safety? I read somewhere on here that doing a single RAIDZ wasn't recommended for performance... How would you guys recommend configuring a server that currently has 4 drives available with the following concerns?
1. Ability to expand later.
2. Best performance for ESXi Datastore.
3. Reliability in the event of drive failures.

 

[danswartz]

1. RAIDZ* can't be expanded per-se. You can add another vdev (raidz2 or whatever) to the pool, but that requires (realistically) 3 drives each time. The advantage of a raid10 setup, is you can add two drives at a time.
2. ESXi tends to do a lot of random reads (AFAIK), due to vhd's for different VMs. raid10 allows ZFS to round-robin the reads, to increase random-read performance.
3. raidz2 with 4 drives is safer than raid10, since the latter has a 33% chance that the 2nd failure takes out your mirror, but unless this is incredibly mission-critical, I wouldn't worry about it - e.g. stick with raid10.

 

[madrebel]

of those 3 options, pick 2.

ability to expand is limited by your ability to add cards for JBOD or the existing server's bays. if you do a 4 drive raidz2 then you can only expand by adding more 4 drive raidz2 vdevs (not 100% true but for best practices this is what you do)

best performance doesn't say much. raidz2 performs well for large sync writes. small random writes, not so much. mirrors will always out perform parity in reads. raidz2 reduces available IOPs to that of a single drive however down the road as you expand with more vdevs your IOPs increases for each vdev, same story for mirrors.

reliability is always determined by how many drives can be lost before your data gets nuked. with mirrors you can in theory lose half your drives and still have available data but are you going to bet the farm that you won't lose both drives in one of your mirrored vdevs and that only one drive in each pair will fail at any given time?

example:

you have 24 drives in 12 mirrors and 24 drives in 6x4 double parity arrays.

in zfs this looks like this
"mirror pool"
mvdev1
mvdev2
mvdev3
mvdev4
mvdev5
mvdev6
mvdev7
mvdev8
mvdev9
mvdev10
mvdev11
mvdev12

"parity pool"
pvdev1
pvdev2
pvdev3
pvdev4
pvdev5
pvdev6

now in both of these scenarios you can lose 12 drives and still have accessible data. however, only with the parity example is this likely to ever be a realistic scenario (and frankly if it happened you're on razor thin ice anyway) as with mirrors sure, you can lose 12 total drives but only one from each vdev where as with double parity you can lose 12 drives and those can be any two in any of the 6 vdevs and you're still in business.

 

[coolrunnings]

Sounds like RAIDZ2 is closest to what I need. Well internally I am out of space on these servers but my HBAs can give me access to more storage if I could find an external hotswap box that I could hook up with mini-SAS cables that would not be an SAS expander... Do they make anything like that? Wonder how hard it would be to do that on my own if needed.

 

[Dalfo002]

If you are looking for non-expander external enclosures you could use single drive eSATA enclosures, or you could get a used duplicator tower and fill it with hot-swap enclosures. If you want a rack-mounted solution just get a rack-mount chassis with hot-swap cages and wire up as many SAS pass-through cables as you need to direct-wire the drives to your HBA.

 

[danswartz]

mirrors won't give you the availability of triple parity.

I wasn't aware that was a requirement here. He was asking about mirrors, and I answered him.

 

[madrebel]

he asked if he would get the same redundancy (which i think he really means availability) as triple parity. i said no, but, i always forget ZFS makes 3, 4 or more way mirroring really simple. so, yes, mirroring can provide great availability but mirroring has significant useable size limitations.