ZFS NFS Performance Question

[dsumike]

I have been trying to run a few performance tests on my recent ZFS setup, but I'm getting strange performance over NFS via ESXi. I don't seem to have any issues from other O/S's (CentOS) when mounting via NFS... so this is curious.

Quick background: I have a 20-disk setup (10k 300gb SAS drives) configured as 10 mirrored vdevs connected via a Supermicro JBOD enclosure that uses built-in SAS expanders. I'm connecting to an LSI 9200-8e running on a 2x E5420 Xeon @ 2.5GHz w/ 16GB ram. OpenIndiana + Napp-It with basically default settings.

bonnie++ results show that the current setup should be able to do at least around 480MB/s writes and 705MB/s reads, so the drives + expanders shouldn't be an issue.

NAME  SIZE  Bonnie  Date(y.m.d)  File  Seq-Wr-Chr  %CPU  Seq-Write  %CPU  Seq-Rewr  %CPU  Seq-Rd-Chr  %CPU  Seq-Read  %CPU  Rnd Seeks  %CPU  Files  Seq-Create  Rnd-Create 
 rpool  544G  start  -                                                 
 storage  2.71T  start  2012.04.04  32G  105 MB/s  98  479 MB/s  94  223 MB/s  53  82 MB/s  96  705 MB/s  67  2651.6/s  7  16  24977/s  13558/s

The slowdown appears to be related to synchronization, but I can't quite figure out how to fix it. When I migrate a Virtual Machine to the NFS share via ESXi, I am seeing what I consider fairly slow performance, see below:

root@storage:~# zfs set sync=standard storage/test
root@storage:~# zpool iostat 2 | grep storage
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
storage      424G  2.30T     10    276  1.13M  27.7M
storage      424G  2.30T      0    746      0  70.5M
storage      424G  2.30T      0    398      0  41.2M
storage      424G  2.30T      0    405      0  41.8M
storage      424G  2.30T      0    491      0  43.5M
storage      424G  2.30T      0    423      0  44.4M
storage      424G  2.30T      0    521      0  40.7M
storage      424G  2.30T      0    412      0  43.1M
storage      424G  2.30T      0    426      0  44.4M
storage      425G  2.30T      0    489      0  43.6M
storage      425G  2.30T      0    355      0  36.8M
storage      425G  2.30T      0    446      0  39.2M
storage      425G  2.30T      0    403      0  41.8M

I am unable to even come close to maxing out a single GbE link. Compare this to if I do a simple: dd bs=8k count=30000 && sync from the same pool mounted from a linux machine, I get the following results:

root@storage:~# zfs set sync=standard storage/test
root@storage:~# zpool iostat 2 | grep storage
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
storage      433G  2.29T      0    495      0  43.9M
storage      433G  2.29T      0    657      0  82.0M
storage      433G  2.29T      0    924      0  92.9M
storage      434G  2.29T      0    988      0   110M
storage      434G  2.29T      0    753      0  94.2M
storage      434G  2.29T      0    816      0   101M

At this point, I am limited only by the network speed, which is what I originally expected.

While not recommended, as a test I tried disabling sycn on the pool to see if that would fix anything. It appears to give me better burst speeds, but has it's own performance quirks:

root@storage:~# zfs set sync=disabled storage/test
root@storage:~# zpool iostat 2 | grep storage
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
storage      419G  2.31T     10    267  1.15M  26.5M
storage      419G  2.31T      0  2.04K      0   250M
storage      419G  2.31T      0      0      0      0
storage      419G  2.31T      0  1.53K      0   192M
storage      420G  2.30T      0    443      0  48.3M
storage      420G  2.30T      0      0      0      0
storage      420G  2.30T      0  2.01K      0   246M
storage      420G  2.30T      0      0      0      0
storage      420G  2.30T      0  1.65K      0   209M
storage      421G  2.30T      0      0  63.9K      0
storage      421G  2.30T      0  1.98K      0   243M
storage      421G  2.30T      0      0      0      0
storage      422G  2.30T      0  1.20K      0   135M

The pool seems to burst write, then lag, then burst write, then lag.

As somewhat of a last-ditch effort, I even tried adding an SSD that I had laying around (256 Crucial M4) as a write cache:

	NAME                       STATE     READ WRITE CKSUM     CAP            Product
	storage                    ONLINE       0     0     0
	logs
	  cxtxd0    ONLINE       0     0     0     256.06 GB      M4-CT256M4SSD2

Yes, it's not an awesome SLC SSD, but it should be better than nothing. However, even with the SSD as a write cache, the performance increase seemed very lackluster from ESXi. The measured IOPS via iostat had the same high/low, but it was higher overall with 200MB/80MB high/low's it appears:

root@storage:~# zfs set sync=standard storage/test
root@storage:~# zpool iostat 2 | grep storage
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
storage      436G  2.29T      0  2.00K      0   182M
storage      436G  2.29T      0  1.21K      0  82.3M
storage      436G  2.29T      0  2.32K      0   216M
storage      436G  2.29T      0  1.24K      0  84.6M
storage      436G  2.29T      0  2.25K      0   229M
storage      437G  2.29T      0  1.42K      0  93.1M
storage      437G  2.29T      0  1.19K      0  81.0M
storage      437G  2.29T      0  2.49K      0   239M
storage      437G  2.29T      0  1.16K      0  79.1M
storage      437G  2.29T      0  2.29K      0   232M

However, I think this is misleading, as it's picking up both the write to the log and to the drives when it pushes out to the drives.

Leaving a basic iftop window open, I saw that I averaged 60-80 MB/s. While this is an improvement over the 40-45 MB/s that I saw without the SSD, it's still worse than the 85-110 MB/s I saw with sync=disabled, and I figured with an SSD that boasts over 40k IOPS, that I would be able to at least see better than ~1k IOPS @ 65MB/s with a write cache.

Are there some tweaks I can do to improve performance? Or is this to be expected and I just had unrealistic expectations?

Any thoughts/suggestions?

 

[_Gea]

mostly as expected.

All async writes are going to RAM.
After a few seconds they are written to disk sequentially and optimized

All sync writes are going to RAM and logged/ commited from a Log/ extra Log device
They are also written to disk sequentially and optimized after a few seconds

So its normal that you see a write lag.
This way of optimized writes is more efficient than continous writing

So if you have applications like ESXi + NFS that always requires sync writes and if you like more performance, your only option is to disable sync demand (can cause a data loss on power loss) or to add a separate Log device with very low latency and high IOP/s, best are DRAM based ones for example like a (expensive) DDRdrive or ZeusRam disc

 

[packetboy]

What are you NFS mount options?

Make sure you are using 'noatime' ... also possible that rsize and wsize options could help read/write perf.

 

[Stanza33]

Yup, add a SSD as a ZIL device and watch it fly...

 

[cantalup]

just a thought.
esxi-nfs without sync is not a good way as I know.
how about esxi-iscsi
the common NFS params as mentioned in this thread for playing around with read size and write size .

 

[dsumike]

mostly as expected.

That's what I was afraid of. Don't get me wrong, performance isn't "terrible" -- I was just hoping for more, especially with higher quality disks.

So if you have applications like ESXi + NFS that always requires sync writes and if you like more performance, your only option is to disable sync demand (can cause a data loss on power loss) or to add a separate Log device with very low latency and high IOP/s, best are DRAM based ones for example like a (expensive) DDRdrive or ZeusRam disc

The DDRdrive and ZeusRam disks are just two expensive for my budget. Are there any other recommendations, like say an SLC SSD that's known to have low latency, that are in the sub ~$500 range that would be worth looking at?

What are you NFS mount options?

Make sure you are using 'noatime' ... also possible that rsize and wsize options could help read/write perf.

I used default settings. On the CentOS box that I mounted the NFS store from, I got excellent performance with the default settings.

In ESXi, I also just went through the "Add Datastore" process, there were no visible options for rsize/wsize.

 

[dsumike]

Yup, add a SSD as a ZIL device and watch it fly...

As you see above, I did add an SSD as a ZIL. I got 'better' performance, but I don't feel like it was screaming fast. ESXi was able to get up to 80-85MB/s averages, but it was still slower when compared to the 95-105 MB/s I was still able to get via the linux dd.

This was accented when I teamed two 1GbE connections and ran two tests from two different hosts simultaneously. The ZIL method seemed to basically still be capped around 90 MB/s, where as the Linux dd got up to around 180 MB/s.

just a thought.
esxi-nfs without sync is not a good way as I know.
how about esxi-iscsi

iSCSI is an option I'm looking at, but we're on ESXi 4.1u1 which is limited to 2TB datastores. We were really hoping for NFS to allow us an easy to manage larger datastore. We run over 250 VM's that are over 50gb each even when thin provisioned.. so it takes quite a few 2TB LUN's to make it work.

If would could trim this up to 2-3 larger NFS datastores, it would help with wasted space that we have to keep available for thin provisioning growth.

the common NFS params as mentioned in this thread for playing around with read size and write size .

For the NFS parameters, is this set on the OpenIndiana side? I didn't really see much for NFS options when adding the datastore to ESXi.

 

[danswartz]

What is the sustained write of your ZIL SSD? Keep in mind, most relatively cheap SSDs are quite a bit slower sustained write vs read.

 

[dsumike]

What is the sustained write of your ZIL SSD?

I used a Crucial m4 256gb for a quick test, connected to a 6Gb port:

• Sustained Sequential Read: Up to 500 MB/s (SATA 6Gb/s)
• Sustained Sequential Write: Up to 260 MB/s (SATA 6Gb/s)
• 4KB Random Read: Up to 45,000 IOPS
• 4KB Random Write: Up to 50,000 IOPS

 

[dsumike]

Do you think a mirrored pair of Kingston SSDNow V+200 60GB 2.5" SATA III Enterprise SSD's would improve performance?

The product claims Up to 460 MB/s (SATAIII) and 4KB Random Write Up to 47,000 IOPS.

Or even a pair of Mushkin Enhanced Chronos 60GB 2.5" SATAIII SSD's?

It claims Sustained Sequential Write of Up to 495MB/s and 4KB Random Write of Up to 85,000 IOPS with a higher MTBF.

 

[Enlightend]

dsumike: did you do firmware updates for that M4?

When I bought my 30 and 60GB ones, I had 450/250 R/W, with firmware updates that became 550/480.

Crucial doesn't use Sandforce and with that, they have to write their own firmware code, took them a while to get a hang of it, but now these drives beat any Sandforce SSD, in advanced setups. (read raided, etc) and their performance doesn't deteriorate.

They don't have the problem that they slow down (both trim and good garbage collection routines), they are blazing fast in R/W with recent firmwares and flashing the firmware is possibly the easiest of all HDD/SSD/Mobo's I've ever done.

Had 8 M4 60GB's in a system, booted the firmware updater and a minute later, it autodetected and flashed all 8 drives.

Remember that for ZIL, you really should use Mirrored pairs of disks, not just 1 or 2 unmirrored ones.
For L2ARC, you can just stripe a bunch of em for hyper read speeds.

 

[_Gea]

I used a Crucial m4 256gb for a quick test, connected to a 6Gb port:

• Sustained Sequential Read: Up to 500 MB/s (SATA 6Gb/s)
• Sustained Sequential Write: Up to 260 MB/s (SATA 6Gb/s)
• 4KB Random Read: Up to 45,000 IOPS
• 4KB Random Write: Up to 50,000 IOPS

These values are "very optimistic"
see www.ddrdrive.com/zil_accelerator.pdf

The problem is, I have not yet seen a really good cheap SSD for ZIL.
So if you need good values with sync write where every write must be committed
until the next can occur, you need high IOP/s. Good sequential values does not matter.

Your options are:
disable sync
disable atime when enabled
enable compress

or add as much mirrored vdevs as possible or
add a really fast ZIL

and
Do not expect data security with checksuns and CopyOn Write without the price.
ZFS is the securest filesystem not the fastest but it scales quite well with vdevs.

and
If you look at iSCSI, its the same when sync write is enabled

 

[dave99]

Remember that for ZIL, you really should use Mirrored pairs of disks, not just 1 or 2 unmirrored ones.
For L2ARC, you can just stripe a bunch of em for hyper read speeds.

Since ZIL removal is available after zpool V.19, I didn't think mirrored ZIL is really a critical item anymore. Important for speed if there is only 1 and it fails, but not so much for data integrity.

 

[MarkL]

Since ZIL removal is available after zpool V.22, I didn't think mirrored ZIL is really a critical item anymore. Important for speed if there is only 1 and it fails, but not so much for data integrity.

Er.. this is incorrect. If your single disk ZIL has a hardware failure then you can lose data.. which is why they recommend mirrored.

 

[dsumike]

dsumike: did you do firmware updates for that M4?

When I bought my 30 and 60GB ones, I had 450/250 R/W, with firmware updates that became 550/480.

That I haven't. It was a quick-test, not a long-term solution... but I will try updating the firmware and trying again. I plan on using this in my desktop once I find a better solution for the ZFS srever, so thanks for the tip!

 

[Enlightend]

If I'm not mistaken, ZIL removal is meant for you to start procedure to remove it, ZFS empties the ZIL cache to disk, then tells you its ok to remove.

If the ZIL disk dies in the middle of writes, you're still in a world of hurt and left with incomplete writes.

 

[patrickdk]

A ZIL is never a single disk, your slog drives can fail, your zil is always in memory, and will be fine.

The issue is if your slog drives fail, AND your system crashs/powerloss/..., and while this shouldn't be that common, a drive going dead can cause all kinds of unknown issues, including causing the system adminitrator to panic and attempt to regain access to the system via a reboot, and therefor will cause data loss.

 

[Enlightend]

That I haven't. It was a quick-test, not a long-term solution... but I will try updating the firmware and trying again. I plan on using this in my desktop once I find a better solution for the ZFS srever, so thanks for the tip!

The original firmware I had on the 60's made the disks hard lock until power cycle after 45xx hours of operation (can't remember the exact number), it was a bug in the firmware code, where it tried to update the SMART counter for hours run, at that specific value and the disk would lock up.

At that time the disks had the R/W performance similar to what you stated.

First I thought the drives died on me, then I started snooping around the web and found that it was a firmware bug, read some more and saw they updated the firmware to increase performance too.

The SMART bug only made the drive lock up until you cycled power, no data was lost.


Did the update, bug was gone and drive speed was crazy.
Did a reset/fill of the drives, re-installed Windows 7 fresh, and drive speed was even more nutters

And I'm running the desktop ones with AMD BIOS Raid, so they don't take advantage of TRIM. Just the background garbage collection. Speed is still the same.

 

[dsumike]

These values are "very optimistic"
see www.ddrdrive.com/zil_accelerator.pdf

The problem is, I have not yet seen a really good cheap SSD for ZIL.

Your options are:
disable sync
disable atime when enabled
enable compress

or add as much mirrored vdevs as possible or
add a really fast ZIL

Thanks. For now, I think I am limited to a cheap SSD and/or disable sync.

I might try compression. Is the default LZJB the most recommended for quick performance gain without taxing the CPU too much?

 

[dave99]

Er.. this is incorrect. If your single disk ZIL has a hardware failure then you can lose data.. which is why they recommend mirrored.

According this post, it's a pretty narrow failure condition that would allow for data loss:
http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg47835.html

My understanding of the ZIL is that it is never read from unless there is a system crash /power failure. In normal operation, if the ZIL fails by itself, the data being written to disk still exists in system memory and will write successfully.

I'm thinking a ZIL with a battery or supercap (acard, ssd with supercap, etc) as an example. I suppose using a consumer SSD you'd be wise to mirror.

 

[apnar]

Also, in the worst case where you lose the ZIL and the machine powers down you only lose data that was in flight (last few seconds of writes). The filesystem itself now remains consistant.

 

[cantalup]

....

Your options are:
disable sync
disable atime when enabled
enable compress


....
and
If you look at iSCSI, its the same when sync write is enabled

disable sync would speed up, but the risk is accelerated ...
disable atime, should be ok
enable compressing, should be ok when processing power is no limited .

the interesting part is NFS 4 versus NFS 3...

---------------
on iscsi, there was white papers on vmware that explained multipath..

------------

how about channel/team bonding 802.3ad lagg?.( need managed or smart switch)
on my experience, lagg helps.
I have freenas 8.0X and centos 6.X with lagg that satisfied me.

---------
many factors ... where we have to pick some of them to fit our expectations.

 

[dsumike]

how about channel/team bonding 802.3ad lagg?.( need managed or smart switch)
on my experience, lagg helps.
I have freenas 8.0X and centos 6.X with lagg that satisfied me.

I didn't really notice any performance with a LAG when sync=standard was set. However, when sync=disabled was set I was able to get a full 200 MB/s (maxing out two 1GbE's) via NFS when transferring from two or more separate hosts though.

Unfortunately, even with 20x 10k SAS drives, it appears my only option is disable sync until I can do more testing with the ZIL/SLOG.

 

[_Gea]

I didn't really notice any performance with a LAG when sync=standard was set. However, when sync=disabled was set I was able to get a full 200 MB/s (maxing out two 1GbE's) via NFS when transferring from two or more separate hosts though.

Unfortunately, even with 20x 10k SAS drives, it appears my only option is disable sync until I can do more testing with the ZIL/SLOG.

Even a 10k disk has only a few hundred IOP/s compared to the thousands or ten-thousands of a SSD

When you look at the price of 10k SAS disks, you may consider SSD only pools
if you need high IOP/s and high sequential transfer rates with multiple users.

(I use Intel 320 in a n * three way mirror config with my performance sensitive VM's)

 

[dsumike]

Even a 10k disk has only a few hundred IOP/s compared to the thousands or ten-thousands of a SSD

When you look at the price of 10k SAS disks, you may consider SSD only pools
if you need high IOP/s and high sequential transfer rates with multiple users.

(I use Intel 320 in a n * three way mirror config with my performance sensitive VM's)

Yeah; I get that. I guess I just didn't realize how much of a performance hit you take with sync=standard. With 20x 10k drives, I have ~1500 write IOPS and 2000+ read. It's not 10's of thousands, but it's still "good".

Is there a an easy to understand explanation of what exactly is happening with sync=standard vs sync=disabled, and more so, how and why this setting when enabled throttles my 200 MB/s LAG down to 40 MB/s?

 

[_Gea]

Is there a an easy to understand explanation of what exactly is happening with sync=standard vs sync=disabled, and more so, how and why this setting when enabled throttles my 200 MB/s LAG down to 40 MB/s?

I would describe as followed:

Problem:
Your server wants to count from one up and write these numbers to disk as fast as possible.

with sync=disabled
All writes goes to RAM for say 5 seconds. Write speed is only RAM limited.
Then after 5 seconds, all disk-heads are positioned to a free area and all numbers
are streamed from RAM to your Pool with sequential write performance of the pool.
I/O is not a limiting problem if there are no other concurrent writes to do.

with sync=enabled
the above is happening as well but additionally:

Each number must be logged and confirmed from pool (log-area) immediatly..
You are hammering your pool with numbers. Speed is limited by I/O performance

I have heard about reports of about 100x speed difference between sync and non sync writes
so your factor 5 is quite small due to your fast disks and limits of your network.

 

[x-cimo]

What kind of ZIL cache device would be able to do 300MB/s + ?

Also what is so different to be able to do 500MB/s + locally but only 80 with NFS?

 

[apnar]

Is there a an easy to understand explanation of what exactly is happening with sync=standard vs sync=disabled, and more so, how and why this setting when enabled throttles my 200 MB/s LAG down to 40 MB/s?

When a program makes the system call to write data it has some choices about exactly how to do it. Normal programs just call a standard write system call, which returns as soon as the data has been transferred to the OS layer for writing. The OS then takes care of writing the data to disk on it's own some time later but in the mean time the program continues on assuming the data has been written.

Some programs (usually ones that really care about consistency, like databases or ESXi) will instead call a synchronous write, which means that the system call is not supposed to return to the program until the data has been physically written to disk. This type of system call is obviously much safer but takes significantly longer as the program has to wait while the data is physically written to disk.

There are some file systems out there that chose to simply lie when a program asks for a synchronous write, they will just return immediately treating the call just as if it was a standard write. ZFS does not lie, if you ask for a synchronous write it will not return until that write has either been written to a ZIL device or to the pool itself. If you change the setting to "sync=disabled" you are telling ZFS to now lie and treat all writes as standard writes.

Also what is so different to be able to do 500MB/s + locally but only 80 with NFS?

The difference is you're not actually testing synchronous writes locally, you're likely just doing a DD. If you whip up a program that writes data but does it via synchronous writes you'll see your local number drop way down.

Alternatively, you could test NFS from another client that doesn't force synchronous writes (as ESXi does), like Linux or Solaris (using specific mount parameters). That should show your performance is limited to your network connectivity as opposed to sync write speed.

 

[computerfixitguy]

Add more RAM. 10x if you can. 16GB just isn't enough.

If thats not a complete enough of an answer keep reading. I have spent tens of thousands wading through the morass of poor zfs performance regarding ESXi that I finally get it.

Add more RAM. As much as you can possibly squeeze into a server. If you have enough L2ARC isn't used, for my ESXi environments its hardly used. But even with 64 GB RAM they were poorly used. I will not do a ZFS setup without 200+ GB RAM anymore. Let me give you sense of where I come from.

I have 200 VM's hitting a 22 spindle mirrored vdev pool (10K disks). I have four of those 22 spindle pools. I use DDRDrive X1's for ZIL (x 2) for throughput. Each zpool gets around 22,000 iops and sustains 300MB/s write throughput because of the 2 x1's. I cannot say enough good things about the DDRDrive or Christopher George.

That being said I am suspecting the X1's are the bottleneck because they are a PCiE 1x device on a single lane so they are limited to 170MB/s (thats why I have two). So I have sitting here some STEC ZeusRAM SAS 8GB RAM modules. Act like RAM but can slide into my SAS chassis. I am excited to see if a 8 lane PCiE 2.x device (LSI 9200-8e) can benefit from the bandwidth of the STEC even though the IOPS might not be as high.

Along the way I added more RAM going from 10 to 32 then 48, finally 64 in the same machine. I bought the DDRDrive and that didn't fix it but it was WAY WAY WAY better. Ultimately had I just started with insane amount of RAM I wouldn't have felt like I was loosing the battle every time.

Another tip: Add more RAM. Just kidding (really add more RAM), break your workloads into similar types. Example: Keep mail servers together because they are all random I/O and will trash your disks the same way. Do not put them on the same disks as a Web server because the web server is mostly reads and the mail is a fewer percentage of reads. Modular is good. I even tried the all SSD zpool path but I found without enough RAM it was a waste of money.

Having a ZIL device is huge. Doesn't have to be big, my ZIL only gets a few hundred megs of data on it ever. Thats why a 4GB DDRDrive works at all. I have Intel X25 160GB for my L2ARC and they are hardly used, only about 10G of data on them at any one time. I finally experienced a device failure on one of those and, it was not awesome. Throughput fell because the SATA device wouldn't just go away. If you can have a couple or at least a spare. I buy used ones on Ebay because it doesn't matter how much life they have left if they are cheap.

I was going to try a Intel X25-M 32GB SLC for ZIL on a less important pool but I bought the ZEUS RAM's instead. This weekend I will have a race between my X1's and my DDRDrives.

After I added enough RAM I noticed that the L2ARC's aren't used at all. After I moved to a mirrored vdev empty pool things improved. You may not know that you can just keep adding disks on a mirrored vdev making the string of disks just longer and longer. Not a good plan for static data. I do that also but only on my backups. As the backups age they remove them selves from the oldest disks in the strand.

Another Vmware specific tip. Keep vcenter off the main device. Put it somewhere else. I have experienced moments where ZFS is being a turd and Vcenter therefore becomes a turd when it doesn't have to.

My favorite places to read on zfs are:
ZFSBuild, http://constantin.glez.de, and http://www.c0t0d0s0.org

All three of them say add more RAM, but it never quite hit home until I did.