ZFS Pool & Drives

[dsumike]

I have an existing server that I want to repurpose as a test system for zfs. I have a 6Gb SAS Enclosure for storing up to 24 SAS drives, and I picked up an LSI 9200-8e to connect it to a spare Dell PE2950.

The intended purpose of this setup is ultimately to provide an NFS datasore to ESXi, with the hopes of being able to expand via a second shelf of 3TB (24x300GB 10k SAS, RAID10) to be added later, totaling 6TB of useable space.

With this in mind, are there any "recommended" mid-priced SSD's that would make a good read cache for roughly 3-6TB of space used as an ESXi datastore?

Would something like a Kingston 240GB "Enterprise" SSD work? The price-point is right, but if it's not a good fit, it doesn't matter. The size is workable, and as you can add multiple cache drives, it would be fairly easy to expand as needed.

I'm just curious if this "Enterprise" drive would hold up to to the read demands of hosting ~30 50GB VM's or so. I've also read a post or two about drive latency being a potential bottleneck. Are there any good resources or guides showing L2ARC options, preferably for various price ranges?

 

[Nex7]

The intended purpose of this setup is ultimately to provide an NFS datasore to ESXi, with the hopes of being able to expand via a second shelf of 3TB (24x300GB 10k SAS, RAID10) to be added later, totaling 6TB of useable space.

With this in mind, are there any "recommended" mid-priced SSD's that would make a good read cache for roughly 3-6TB of space used as an ESXi datastore?

Careful with L2ARC -- it requires ARC to reference it. For this reason, I tend to recommend maxing RAM out (to at least 8 GB DIMM's in every slot, maybe not 16 GB DIMM's due to price) before even considering any L2ARC. An NFS use-case with defaults will end up using 64-128k average block size, which means each 100 GB of L2ARC is going to require between 140-280 MB of ARC(RAM), which is not bad but do keep it in mind. Also, when doing L2ARC, MANY SSD's take massive performance dump when they are more than 80-90% utilized (kinda like ZFS itself!), so you'll want to partition them to only use 80% or so.

I'm just curious if this "Enterprise" drive would hold up to to the read demands of hosting ~30 50GB VM's or so. I've also read a post or two about drive latency being a potential bottleneck. Are there any good resources or guides showing L2ARC options, preferably for various price ranges?

Most decent SSD's will outperform spinning media quite handily (assuming you stay within their realm of happiness, like the above note about going over 80% on them). I want to touch on the latency and 'round-robin' talk I saw a bit earlier in the thread though. The ZIL round-robins between log vdevs; the L2ARC does not. Or, I should say, it does not for reads.. as it can't, if you think about it. I'd have to go check the code or ask a few coworkers to figure out if L2ARC writes are round-robin'd or if they're written based on more intelligent logical choice, but once they're ON a 'cache' vdev, reads have to go to THAT vdev.. I'm fairly certain (not in the mood to go code spelunking, but I'd think it would have come up by now if it were so) ZFS is not "round-robinning" reads from L2ARC, I'd wager they're read on demand from whatever cache vdev has the data (as determined by the ARC pointer entry), so read & write 'latency' from disks in an L2ARC vdev are likely not going to have the same bottlenecking affect that they do on log vdevs.

And similarly, if you did two raidz2 vdevs with 12 disks in each, then you would have write IOPS as two disks (two vdevs), but read IOPS as 24 disks (the number of individual disks)?

No. RAIDZ does not operate in the same manner as mirror. I can't do 'more reads' on RAIDZ like I can on mirror vdevs. How RAIDZ is going to handle your reads/writes has a lot to do with just how many spindles are in the vdev and the average block size of the writes, but the general rule of thumb for sizing is 1 vdev of raidz = 1 vdev of IOPS, both read or write.

 

[dsumike]

10krpm sas drives give around 145iops (seagate 10k 2.5=149, seagate 10k 3.5=147)

That gives 1740iops writes, and double that for reads.

So I just got my OpenIndiana + Napp-It install up and running this afternoon. I figured that I would run a few performance tests as I added vdev's to see where performance begins to plateau. Starting with 2 mirrored vdev's, I ran bonnie++ via napp-it, and while it's running I am seeing the iostat results below:

# zpool iostat -v storage
                              capacity     operations    bandwidth
 pool                       alloc   free   read  write   read  write
-------------------------  -----  -----  -----  -----  -----  -----
storage                    32.0G   524G    129    262  16.2M  23.6M
  mirror                   16.0G   262G     64    131  8.08M  11.8M
    c6t5000C5003B69133Bd0      -      -     32    119  4.04M  11.8M
    c6t5000C5003B6C6E07d0      -      -     32    119  4.04M  11.8M
  mirror                   16.0G   262G     65    131  8.09M  11.8M
    c6t5000C50039ADDA4Fd0      -      -     32    119  4.05M  11.8M
    c6t5000C50039B238FBd0      -      -     32    119  4.04M  11.8M
-------------------------  -----  -----  -----  -----  -----  -----

Yes, I realize this is a small pool to start with, 2 mirrored vdevs. I'm just curious how this compares to expectations. The write speed doesn't seem like it's anything to brag about, but it could be the test bonnie++ was running. The IOPS for 10k SAS2 drives seems to be about as expected, ~151 (32r+119w).

Does this seem about right?