ZFS Build Question

[0x736967]

I am looking to build a ZFS based storage solution using OpenIndiana for home and had a couple of questions. I will be using 6*3 TB drives (heterogeneous but they are all 7200RPM, SATA 6Gb/s, 62MB Cache), ASRock C226 Motherboard, Xeon E3-1220, and 8GB of ECC DDR3 RAM. The storage space will be used for personal files (500GB or so), Surveillance Cameras (3TB set to record on motion 24*7*365 with auto delete), and to host VMs(Home lab which will take about 500GB to expose to ESXi using NFS/iSCSI). I had a few questions with regards to the setup.

1. Should I do a Raidz2 pool or two way mirror of 3*2(3TB Drives) pool? In raidz2 I will have close to 12TB of usable space with a two disk redundancy and in the mirror configuration I would get 9TB of usable space. I read that the mirror configuration will give me better performance so I was leaning towards doing that instead.

2. I was planning on using a separate hard drive for the OS but then I read that I can increase the performance of a pool by using an SSD for the secondary cache. If I get an SSD can I install the OS on a small partition and create a secondary partition for the pool to use as a cache? Will this work or will the pool need an entire SSD dedicated for caching? This leads to another question which is should I use the conventional hard drive and spend the rest of the money to upgrading the server to 16GB of ECC RAM? What gives me more performance here?

3. Deduplication and compression are two features that I am not sure about if I should be using or not. I am thinking that my VMs will benefit from deduplication since they all will be running the same OS with a few dissimilarities here and there but otherwise they should deduplication well. I am not sure about compression and the type of performance hit that I will be taking if doing so. I am thinking that the processor should be able to handle it without any issues.

Thanks for all the help. This is my first forum post and I hope that I didn’t screw anything up. This seems like a good community.

 

[iroc409]

1. Mirrors would probably be better from a risk standpoint and performance, from some of what I've been reading lately.

2. Using a cache device may or may not provide any performance gains, depending on a lot of things. I'd personally probably go to 16GB RAM first, but others here can talk more authoritatively about this.

3. Deduplication takes a massive amount of overhead. You'd need 16GB of RAM to even try it on your system, as it (by old rule of thumb) needs at least 1GB per TB of disk storage to run. Typical wisdom says don't run dedup unless you have a lot of horsepower and RAM, but especially if the storage server itself is a VM (and not just the host OS), I think you'll run into unpleasantness.

 

[danswartz]

For random I/O, you really want mirrors. raidz* only gives 1 drive random IOPs per vdev. I wouldn't bother with a cache ssd - 16GB as jroc said would help a lot. Do NOT use dedup!

 

[patrickdk]

the whole 1gb per tb to dedup only holds true if you use your pool to store large files, and at the cost of having horrible dedup rates.

If you want dedup to work, you want smaller block sizes, and that means 32times as much ram used.

doing raidz2 will be good for your video cams, but horrible for vm usage. Doing mirrors will help vm usage, but waste a lot of space for your cams. Using a ssd cache here, could help speed up vm read performance for a raidz2, but writes will still suffer.

Compression you should always just leave on, just set to lz4 or lzjb, if you have something you don't write to often, you could set that to gzip, but gzip will limit you to aprox 25MB/sec per cpu core.

Personally if I was doing this, I would just raidz2 the drives, and use them for cams, and maybe personal files.

then mirror two ssd's for the vm's

 

[rsq]

Welcome to the forums 0x736967.

Lots of useful information can be found here:
http://nex7.blogspot.be/2013/03/readme1st.html
Especially the sections on dedup and ARC/L2ARC are interesting for you.

More in-depth info about ARC/L2ARC here:
http://nex7.blogspot.be/2013/10/lots-of-l2arc-not-good-idea-for-now.html

Personally, I would not bother with cache devices, I would invest in more memory, that way, the blocks used frequently (like VM OS files) would be served from the ARC out of RAM.

You can also maximize the usefulness of your RAM by installing a base VM on one ZVOL, and then clone that ZVOL for each VM. This will save you space on disk, and the blocks for the OS files will be hit more frequently and remain in the ARC in RAM.

Top tip: if you make a ZVOL for a VM, use the -s option. That way you can oversubscribe the amount of storage you have. This avoids that you have to set very narrow margins on your VM disks. If the volumes then grow, you can just add storage as needed. (If you want to add storage, using mirrors is best, because then you only need to add 2 drives at a time.)

 

[0x736967]

From what I gathered here I should spend the extra money on more RAM and not run dedup. I am leaning more towards the mirroring option since it will give me better performance for VMs at an extra cost of storage space for camera video. I don't think I will ever use more than 4TB honestly so storage space is not an issue and it is also cheap.

Depending on the performance of VMs I might eventually get two SSDs and mirror those to create another pool for VMs. Are there any recommendations with regards to SSDs? Is there a specific controller that would be better for this purpose? Thanks for all the comments, they have been very helpful.

 

[spazoid]

You could consider running two of the disks in a mirror for the VM's and the remaining 4 drives in a RAID-Z. That'll give you relatively good performance for your VM's, and "waste" one less disk overall.

I'd wager that with 16 GB of RAM (which you should absolutely upgrade to) will be able to serve most of your VM IO directly from ARC.
I've attached a copy/paste from arcstat running on my omnios machine which is serving an esxi server running 8 VMs. As you can see, even with just 12GB of ARC I get a 0% miss rate, and this is true for most of the time. Only during boot of the VMs do I see more than 20% miss rate. This isn't a busy setup, but performance is great, and the few IO's that do miss the ARC are served with very acceptable performance from a 4 disk mirror on old 7200 rpm disks.

time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c
10:13:16 5 0 0 0 0 0 0 0 0 12G 12G
10:13:17 107 0 0 0 0 0 0 0 0 12G 12G
10:13:18 19 0 0 0 0 0 0 0 0 12G 12G
10:13:19 38 0 0 0 0 0 0 0 0 12G 12G
10:13:20 0 0 0 0 0 0 0 0 0 12G 12G
10:13:21 0 0 0 0 0 0 0 0 0 12G 12G
10:13:22 0 0 0 0 0 0 0 0 0 12G 12G
10:13:23 38 0 0 0 0 0 0 0 0 12G 12G
10:13:24 0 0 0 0 0 0 0 0 0 12G 12G
10:13:25 0 0 0 0 0 0 0 0 0 12G 12G
10:13:26 0 0 0 0 0 0 0 0 0 12G 12G
10:13:27 112 0 0 0 0 0 0 0 0 12G 12G
10:13:28 13 0 0 0 0 0 0 0 0 12G 12G
10:13:29 38 0 0 0 0 0 0 0 0 12G 12G
10:13:30 0 0 0 0 0 0 0 0 0 12G 12G
10:13:31 0 0 0 0 0 0 0 0 0 12G 12G
10:13:32 6 0 0 0 0 0 0 0 0 12G 12G
10:13:33 13 0 0 0 0 0 0 0 0 12G 12G
10:13:34 38 0 0 0 0 0 0 0 0 12G 12G
10:13:35 15 0 0 0 0 0 0 0 0 12G 12G
10:13:36 1 0 0 0 0 0 0 0 0 12G 12G

I did have a Samsung 840 for L2ARC, but removed it because it rarely got any hits at all, so I would skip it initially and buy one if you see lots of ARC misses.

 

[0x736967]

spazoid, thanks for the detailed information. I am still waiting for my disks to arrive so I haven't done any configuration yet. From what I gathered, creating two separate pools for VMs and another for Data might be a better option rather than having a single pool. Do you have any numbers with regards to the disk latency that you are seeing from your hypervisor?

I am trying to decide whether to use SSDs or conventional drives for the VMware pool since I will be having two ESXi hosts accessing the LUNs on the pool. Any thought with regards to hybrid drives instead of SSDs?

 

[spazoid]

I would personally never use hybrid drives, especially so for ZFS. I doubt very much that the on-disk logic would be smarter than letting ZFS handle caching.

The latencies I see vary a lot, since it's either a hit in RAM (ARC) or slow-ish 7200 RPM disk. Average according to the vSphere client for the last hour is 1.8 ms read and 0.2 ms write latency.

If you can afford it and really need the performance more than the space, go for SSDs for your VMs. They'll love you for it. I haven't been able to justify it yet, as the performance I get is far better than what I need, and 1 TB of mirrored SSD is still too expensive to spend money on just for the fun of it.