VMware on ZFS speed test results

W

wheelz

Weaksauce
Joined
Feb 4, 2011
Messages
100
I now have my ZFS server up and running and I thought it would be good to share some of my numbers. I will add more scenarios as I have time. My server is running on the following hardware:

1 x Chenbro RM23212 2U 12 bay chassis
1 x Chenbro 84H323210-004 SATA2/SAS Backplane
1 x 820W Redundant Psu 3Y 1+1
1 x Supermicro X8DT6-F
1 x Intel Quad Core Xeon E5606 2.13 Ghz
1 x Intel Thermal Solution STS100C Processor Cooler
3 x Samsung Dual Rank 8GB ECC RDIMMs
1 x PCI Riser ARC2-733-3X8
2 x Intel Dual GB PCIe NIC
2 x SFF 8087 to sata breakout cable
2 x Super Talent 16 GB MLC 7-pin SATA DOM (mirrored for OS)
4 x Seagate Barracuda 1 TB SATAII 300MB/s 7200rpm HD (existing drives for data)
1 x Seagate Constellation ES 1 TB SATAII 300 MB/s 7200rpm HD (added new data drive)
No SSD *Yet* :)

It is running Solaris 11 Express with napp-it on top (thanks _Gea!). I have the network configured with 2 x 2 bonded GB NICs (LACP - each to a separate switch) for iSCSI or NFS traffic and 1 onboard GB NIC for administration.

For reference, here is my Bonnie results:
File - 32G
Seq-Wr-Chr 75 MB/s %CPU 99
Seq-Write 337 MB/s %CPU 42
Seq-Rewr 161 MB/s %CPU 28
Seq-Rd-Chr 68 MB/s %CPU 92
Seq-Read 358 MB/s %CPU 21
Rnd-Seeks 818.9/s %CPU 3
Files - 16
Seq-Create 26273/s
Rnd-Create 24824/s

Other than that I have only done very simple tests of cloning a 20 GB VM with no compression, dedupe, or encryption so far. I am including my old storage server so you know what I am switching from (Openfiler - I won't bore you with the hardware on that piece of junk).

Old Storage Box to Local Host Disk - 26 Min or 13 MB/s or 105 Mb/s
Local Host Disk to ZFS Volume iSCSI Target (64KB block size) - 6.5 Min or 52.5 MB/s or 420 Mb/s
Local Host Disk to ZFS File iSCSI Target - 6.5 Min or 52.5 MB/s or 420 Mb/s
Local Host Disk to ZFS NFS Share (only uses 1 pair of NICs) - 22.5 Min or 15 MB/s or 121 Mb/s
Local Host Disk to ZFS NFS Share (only uses 1 pair of NICs) with sync disabled (use at your own risk) - 3 Min or 114 MB/s or 910 Mb/s

Compression on
Local Host Disk to ZFS Volume iSCSI Target (64KB block size) - 7.5 Min or 45.5 MB/s or 364 Mb/s
Local Host Disk to ZFS File iSCSI Target - 7.5 Min or 45.5 MB/s or 364 Mb/s

Dedup on
Local Host Disk to ZFS Volume iSCSI Target (64KB block size) - 8 Min or 42.7 MB/s or 341 Mb/s
Local Host Disk to ZFS File iSCSI Target - 8 Min or 42.7 MB/s or 341 Mb/s

Dedup verify
Local Host Disk to ZFS Volume iSCSI Target (64KB block size) - 12 Min or 28.4 MB/s or 228 Mb/s
Local Host Disk to ZFS File iSCSI Target - 10 Min or 34.1 MB/s or 273 Mb/s

Compression & Dedup on
Local Host Disk to ZFS Volume iSCSI Target (64KB block size) - 9.5 Min or 35.9 MB/s or 287 Mb/s
Local Host Disk to ZFS File iSCSI Target - 9 Min or 37.9 MB/s or 303 Mb/s

Compression & Dedup on with Jumbo Frames Enabled and 14 other VMs running
Local Host Disk to ZFS Volume iSCSI Target (64KB block size) - 8 Min or 42.7 MB/s or 341 Mb/s
*this gave a nice boost to help offset the compression & dedup performance loss*

What strikes me is the NFS performance. I had heard that NFS can actually perform better than an iSCSI target for VMware so I am not sure if I am doing anything wrong. Any ideas?

I hope to do some more testing with compression, dedupe, and maybe encryption turned on. I welcome comments, requested tests (though I may only do what time permits), and suggestions.
 
Last edited:
I'm not sure I believe the 'nfs can be faster', if both are tuned right. Compression and dedupe are not likely going to speed this up - this is a network protocol issue, I think. Try setting sync=disabled on the share and retesting nfs writes...
 
danswartz said:
I'm not sure I believe the 'nfs can be faster', if both are tuned right. Compression and dedupe are not likely going to speed this up - this is a network protocol issue, I think. Try setting sync=disabled on the share and retesting nfs writes...
Click to expand...

That certainly made a difference. This is the description in the napp-it interface:

sync=disabled
Synchronous requests are disabled. File system transactions
only commit to stable storage on the next DMU transaction group
commit which can be many seconds. This option gives the
highest performance, with no risk of corrupting the pool.
However, it is very dangerous as ZFS is ignoring the synchronous
transaction demands of applications such as databases or NFS....
Click to expand...

That makes me a little worried as I am fairly newb to NFS shares. I guess I do not know the implications of this in my scenario.

Also adding compression/dedupe/encryption isn't so much to increase performance but to see what the impact of adding these features are on my setup.
 
Also how do other people make the links redundant for NFS? Just use DNS round robin? Right now I can only point my hosts at one of my 2 IPs (1 IP for each bonded pair of NICs).
 
Usually vmware folks use nic teaming or somesuch. If you do the same on the zfs box, it's all transparent... It all depends on what you are writing as to how unsafe you think it is. The problem is that vmware does nfs sync mode with no way to change that, which is why I use iSCSI for datastores. OTOH, I think the danger is overstated, if you have the zfs server on a ups...
 
btw, you said 'that made a difference', but you didn't tell us how much it made?
 
danswartz said:
Usually vmware folks use nic teaming or somesuch. If you do the same on the zfs box, it's all transparent... It all depends on what you are writing as to how unsafe you think it is. The problem is that vmware does nfs sync mode with no way to change that, which is why I use iSCSI for datastores. OTOH, I think the danger is overstated, if you have the zfs server on a ups...
Click to expand...

I have two switches for redundancy so I cannot (at least with my switches) team all 4 NICs. It is on a UPS.. hmm... decisions, decisions... :)
 
Is it possible to have an active/active IPMP (IP Multi-pathing) for NFS? Basically have my two bonded NICs pairs have the same virtual IP address?
 
Starting to get out of my area (still learning vmware myself). Might want to take this to the virtualization board - more than one person there with lots of vmware experience.
 
danswartz said:
Starting to get out of my area (still learning vmware myself). Might want to take this to the virtualization board - more than one person there with lots of vmware experience.
Click to expand...

Oh, I was talking about on the solaris zfs server side. I have the VMware stuff worked out. It is Solaris/ZFS that I am new to. :)
 
But my point was that I think you need both ends to be using the same algorithm, no? I had no idea how you had the vmware side configured...
 
wheelz said:
Oh, I was talking about on the solaris zfs server side. I have the VMware stuff worked out. It is Solaris/ZFS that I am new to. :)
Click to expand...

Yes, IPMP is fully supported on Solaris. See here for one guide: http://solaris-howto.blogspot.com/2008/03/ipmp-on-solaris-10.html. Its a Solaris 10 guide but it all still works exactly the same on Solaris 11 Express. Google "IPMP Solaris" and you'll find dozens more.

Why the interest in IPMP vs a layer-2 solution like LACP? IPMP is subject to packet order problems that can wreak havoc on file sharing protocols like Samba and NFS.
 
PigLover said:
Yes, IPMP is fully supported on Solaris. See here for one guide: http://solaris-howto.blogspot.com/2008/03/ipmp-on-solaris-10.html. Its a Solaris 10 guide but it all still works exactly the same on Solaris 11 Express. Google "IPMP Solaris" and you'll find dozens more.

Why the interest in IPMP vs a layer-2 solution like LACP? IPMP is subject to packet order problems that can wreak havoc on file sharing protocols like Samba and NFS.
Click to expand...

Well right now it seems I can push to NFS a lot faster than iSCSI (although I am open to any tweaks like what danswartz alluded to). However with my 2 switch configuration, it seems like I can only hit 2 GB max (1 bonded NIC pair). I was looking for a way for my NFS datastore to take advantage of 4 GB (both bonded NIC pairs).
 
In our VMware setups we use stacked switches with channel bonding to provide the added bandwidth and redundancy. It sounds like your switches won't stack though, so you might just want to add the NFS share twice to VMware, once on one IP and once on the other (or just use two different shares). Then just split your VMs between the two. You really aren't getting more then 1Gb per ESX server anyway, even if you could get all 4 bonded.
 
Well, I had a good conversation with someone who designs SAN/VMware solutions and it seems like disabling sync may not be the best thing to do. He sited an expert from VMware who basically said, yes you can do it if all you care about is speed and are not really concerned at all with data consistency. So I guess this puts the iSCSI back on top for now...
 
I added some more basic tests using the same 20 GB VM (in my first post). It is interesting how the different features affect the transfer times.
 
My personal view if you need best NFS performance

- use really fast disks
- add a ssd or better a dram write cache drive

BUT
Best is to use a SSD only pool without any cache drive.
Mostly, you want performance. Terabytes are needed for backup only

My NFS VMware datastore server are 4 x 3 way mirror or 5 x 3 way mirror with 120 GB MLC Sandforce.
Performance is great, reliability is below desktop drives, (reason for 3x mirror), and I added 2 spares.

After the 3 years of warranty, i will replace them all, so SLC highend drives are not needed. These lowcost SSD
are not cheap, but they are worth the money. (I would prefer current Intels now)

>> you can feel the difference, no need to make benches
>> I will never go back to spindels.


Gea
 
_Gea said:
My personal view if you need best NFS performance

- use really fast disks
- add a ssd or better a dram write cache drive

BUT
Best is to use a SSD only pool without any cache drive.
Mostly, you want performance. Terabytes are needed for backup only

My NFS VMware datastore server are 4 x 3 way mirror or 5 x 3 way mirror with 120 GB MLC Sandforce.
Performance is great, reliability is below desktop drives, (reason for 3x mirror), and I added 2 spares.

After the 3 years of warranty, i will replace them all, so SLC highend drives are not needed. These lowcost SSD
are not cheap, but they are worth the money. (I would prefer current Intels now)

>> you can feel the difference, no need to make benches
>> I will never go back to spindels.


Gea
Click to expand...

I love that idea... unfortunately the budget does not permit it. I am looking at VM performance because I am assuming this will give my server the highest load, but I will have large file data sets on here as well. I hope I can make this work for both data types now and perhaps some day move the VMs to their own SSD pool.
 
wheelz said:
I love that idea... unfortunately the budget does not permit it. I am looking at VM performance because I am assuming this will give my server the highest load, but I will have large file data sets on here as well. I hope I can make this work for both data types now and perhaps some day move the VMs to their own SSD pool.
Click to expand...

I know the problem about money :))

in our former config we have had 4 Windows AD Server (primary and 3 backup).
First AD-Login of our students could last up to 4 minutes.......................................

Now all four AD server are on SSD pools.
First login of a user to a Pool machine is usually below 30s.

Gea
 
So with my speed tests I thought I would also see how much space I save from using compression and dedup. I cloned a number of VMs from my current storage to the zfs box. The compression rate shows up as 1.13x and the dedup only shows 1.02x. This seems a little low to me since there are a number of VMs with the same OSes. I suppose the bulk of the data is not OS but I somewhat expected a little better ratio. However I then looked at how much it was taking up on the old storage - 668.4 GB and the zfs box only used 508 GB. That works out to ~1.32x which is a little better but nothing to write home about. Hmm... so what it shows in the interface doesn't appear to be accurate.

What is other people's experience with VMs on this? What ratios are you seeing?
 
The dedup ratio you see might be for the whole pool? If you only have one folder/share doing this, that could explain it. I would not use dedup unless you really know you need it.
 
FWIW, I have dedup enabled on only a tiny subset of my pool (4 TB allocated, 96 GB allocated on the 1 folder with dedup enabled) and my pool shows a dedup ratio of 2.0x. So while it does seem to be showing the dedup ratio for the entire pool, it is for the entire pool in regards to the folders that have dedup enabled (in my case the picture folders from my laptop, and my partner's laptop, which I'd expect to be almost 100% duplicated).
 
Last edited:
I am not planning on using dedup for all my data, but just to test I copied the VMs. Right now this is the only data on the pool.

Since the dedup ratio was so low, I thought perhaps if I deleted the biggest actual data holders like a file server. At that point I would expect most of each VM to mainly be the OS.

What is interesting is that even a day after the deletion, the interface doesn't show that this space was freed up. My ESXi box sees only 153.6 GB used, however on my storage box (again only data on the pool) still shows 622 GB allocated at the pool level, 512 GB used at the pool level, and 508 GB used at the folder/volume level. My dedup and compression ratios remain the same. I'm not sure what the difference between allocated and used is, nor why there is the discrepancy... (note no snapshots exist yet).
 
Last edited:
I finally had a change to enable jumbo frames and run my test again to see what kind of difference that makes. Now this isn't a true apples to apples comparison because I now have 14 VMs running on the storage where my previous test had no other VMs running. However that makes it even more of a substantial jump:

Compression & Dedup on with Jumbo Frames Enabled
Local Host Disk to ZFS Volume iSCSI Target (64KB block size) - 8 Min or 42.7 MB/s or 341 Mb/s

That helped off-set some of the speed loss of enabling compression and dedup (by almost 20% even with the extra VMs running on it!).
 
You must log in or register to reply here.
Back
Top