I have been scratching my head for the last 8 hours when benchmarking our storage performance from our xenservers and VMs running on them.
The xenserver uses NFS for storage SR to a ZFS OmniOS machine with 32GB RAM and 8x Crucial M500 SSDs in a raid-z2 configuration.
The connection is 1 Gbit/s to the SAN.
I use sysbench, crystalDisk and dd to measure the performance. NFS sync writes are off during these tests.
The following list of commands were used dd:
Create 10 Gbyte file of random numbers: dd if=/dev/urandom of=newfile bs=1M count=10000
Read test: dd if=/dev/urandom of=newfile bs=1M count=10000
Read/Write test: dd if=newfile of=newfile2 bs=1M count=10000
Write test: dd bs=1M count=10000 if=/dev/zero of=newfile3
Sysbench:
sysbench --test=fileio --file-total-size=10G --file-test-mode=seqrd prepare
sysbench --test=fileio --file-total-size=10G --file-test-mode=seqrd run
sysbench --test=fileio --file-total-size=10G --file-test-mode=seqwr --max-time=300 run
dd locally from xenserver terminal to nfs mounted storage:
- write: 124 MB/s
- read: 120MB/s
- read/write: 57 MB/s
dd from Ubuntu PVHVM VM running on the xenserver:
- read: 90 MB/s
- read/write: 40MB/s
- write: 106MB/s
Sysbench from the Ubuntu VM:
Sequential read (seqrd): 75 MB/s
Sequential write (seqwr): 116MB/s
Crystaldisk from Windows 2008 R2 VM:
My conclusion here is that xenserver is limiting the disk performance for the VMs running on it since the NFS storage performance is good when running tests from the xenserver terminal on the nfs mounted directory.
My questions:
1. Do any one know why (and how I can improve) the disk performance? Primarily in the Windows VM since the result there is really bad.
2. How can the read performance be worse than the write performance? I would have expected it to be the other way around.
Thank you for any input on this!
Note: I have tested different tcp receive and send window sizes (from the default), with Generic Receive Offload (GRO) on and off, Large Receive Offload (LRO). GRO was on (per default in xenserver 6.5) and turning it off reduced performance. I have checked the nfs server nfsstats and nothing odd in there. Iperf shows ~900MBit/s in both directions between UbuntuVM and the san.
The xenserver uses NFS for storage SR to a ZFS OmniOS machine with 32GB RAM and 8x Crucial M500 SSDs in a raid-z2 configuration.
The connection is 1 Gbit/s to the SAN.
I use sysbench, crystalDisk and dd to measure the performance. NFS sync writes are off during these tests.
The following list of commands were used dd:
Create 10 Gbyte file of random numbers: dd if=/dev/urandom of=newfile bs=1M count=10000
Read test: dd if=/dev/urandom of=newfile bs=1M count=10000
Read/Write test: dd if=newfile of=newfile2 bs=1M count=10000
Write test: dd bs=1M count=10000 if=/dev/zero of=newfile3
Sysbench:
sysbench --test=fileio --file-total-size=10G --file-test-mode=seqrd prepare
sysbench --test=fileio --file-total-size=10G --file-test-mode=seqrd run
sysbench --test=fileio --file-total-size=10G --file-test-mode=seqwr --max-time=300 run
dd locally from xenserver terminal to nfs mounted storage:
- write: 124 MB/s
- read: 120MB/s
- read/write: 57 MB/s
dd from Ubuntu PVHVM VM running on the xenserver:
- read: 90 MB/s
- read/write: 40MB/s
- write: 106MB/s
Sysbench from the Ubuntu VM:
Sequential read (seqrd): 75 MB/s
Sequential write (seqwr): 116MB/s
Crystaldisk from Windows 2008 R2 VM:
My conclusion here is that xenserver is limiting the disk performance for the VMs running on it since the NFS storage performance is good when running tests from the xenserver terminal on the nfs mounted directory.
My questions:
1. Do any one know why (and how I can improve) the disk performance? Primarily in the Windows VM since the result there is really bad.
2. How can the read performance be worse than the write performance? I would have expected it to be the other way around.
Thank you for any input on this!
Note: I have tested different tcp receive and send window sizes (from the default), with Generic Receive Offload (GRO) on and off, Large Receive Offload (LRO). GRO was on (per default in xenserver 6.5) and turning it off reduced performance. I have checked the nfs server nfsstats and nothing odd in there. Iperf shows ~900MBit/s in both directions between UbuntuVM and the san.
Last edited: