ZFS ZIL write cache device Ok to use virtualized hdd disk?

cbutters

Gawd
Joined
Dec 30, 2005
Messages
514
I have a ZFS RAIDZ2 created out of 10x2TB hard drives using ESXi on an Intel 80GB SSD as a boot drive with OpenIndiana w/ passthrough storage controller hardware. The pool is v28

Performance has been adequate while copying files to and from the server, 90-110MB per second.. Great... but then I had problems.

The Problem
However once I tried to use utorrent, I ran into some game changing write performance issues... It seems that the synchronous writes that utorrent does while writing data to the ZFS pool over the network are very slow writes for ZFS to handle... utorrent would hang on writes, making the download speed of the torrent drop to almost nothing, and upon completion of the file, it would say "flushing to the disk" and take about 5 minutes to write a 500MB file to the pool.

This was unacceptable to me since it was slowing my downloads down to take hour rather than a few minutes.

My Solution
Doing some reading, I realized I needed a ZIL device (ssd write cache) in order to improve the write performance. I don't want to use up my hot swap bays since I will need them for future expansion, so I looked into USB 3.0 devices, then realized that was a bad idea, I finally came up with the following solution: I created a 8GB virtual HDD device on the INTEL boot SSD using ESXi, and passed it to openindiana and added it as a vdev to the pool as a SSD write cache.

My performance in utorrent and other synchronous writes immediately seemed "normal" after adding the device! my utorrent never dropped download speed and flushing to the file system appeared to be immediate! Other file copies to the system also seem to be much more predictable and smoothed out.

So I am presenting this as either a tip for other home users who might run into this problem, but also since I am new to ZFS, what I am doing is possibly a very stupid idea.... Please feel free to correct me if this is a bad idea, but so far it appears to have worked quite well. Is 8GB adequate, or is less or more better? would it be a good idea to do the same thing and pass another 8GB as l2arc (read cache)?
 
Last edited:
Is that one of the earlier intel g1/g2 ssd's? If it is, I don't believe it has any inherent gc, and as such I'd be a little hesitant with its longer term reliability for this kind of task. But I'm sure someone else can chime in with whether or not it's fine.
 
If your problem was partially solved with adding a ZIL, you can also disable sync write on the dataset you use for torrent. Concurrent sync write demands with small files can slow down performance extremely. If you disable sync, all writes are going to RAM only and are written to disk after a few seconds at once and sequentially with max performance, instead of doing a commited disk write after each block until the next can occur. With a lot of concurrent small writes, all sync writes do the same: wait for disks.

Up to a few seconds dataloss of last data from RAM, waiting for disk write seems not a problem with this usecase when a powerloss happens.
 
omg I have the same problem and had no idea why. uTorrent "Disk Overloaded" ... i'll have to try some of these fixes
 
If your problem was partially solved with adding a ZIL, you can also disable sync write on the dataset you use for torrent. Concurrent sync write demands with small files can slow down performance extremely. If you disable sync, all writes are going to RAM only and are written to disk after a few seconds at once and sequentially with max performance, instead of doing a commited disk write after each block until the next can occur. With a lot of concurrent small writes, all sync writes do the same: wait for disks.

Up to a few seconds dataloss of last data from RAM, waiting for disk write seems not a problem with this usecase when a powerloss happens.

I guess this doesnt work for me in my case because i have one big dataset so I cant disable sync on just the torrent folder .. but then again for my home use, disabling sync wont hurt the rest of the data i store in the dataset either since it's mostly a download/media server
 
pitne, is there a reason the torrent folder is not a dataset? Remember there is not some low limitation on number of datasets. Make a new dataset at the same folder level as the current torrent folder, move all the data from the torrent folder to it, remove the torrent folder, rename the new dataset to the torrent name and set sync=disabled off on it. Done.
 
pitne, is there a reason the torrent folder is not a dataset? Remember there is not some low limitation on number of datasets. Make a new dataset at the same folder level as the current torrent folder, move all the data from the torrent folder to it, remove the torrent folder, rename the new dataset to the torrent name and set sync=disabled off on it. Done.

Oh ok I didnt know I could do that. I guess the only issue then would be instant move operations over CIFS wont work anymore
 
True, a tradeoff. Basically, you can have (effectively) unlimited datasets, and they look just like folders in the hierarchy.
 
It should. Note that the default is sync=standard, so from a clean slate, any child datasets should inherit from the top-level.
 
I realized I needed a ZIL device (ssd write cache) in order to improve the write performance. I don't want to use up my hot swap bays since I will need them for future expansion, so I looked into USB 3.0 devices, then realized that was a bad idea

Depending on your setup, maybe you could utilize one of these:


If you have a spare expansion slot that's of no use to you (maybe an unused PCIe x1 or a regular old PCI slot), you could turn the slot into a hot-swap SSD bay. :)
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
So just to clarify, in this usage case, are we saying it is best to do:

A) do not use a ZIL device and turn sync off for a dataset
B) use a ZIL device AND turn sync off for a dataset
C) use a ZIL device and leave sync enabled

what are the pros and cons of each?
 
In general, you don't want sync=disabled unless you don't mind losing data on a write. Most apps on the ZFS unit itself will not be doing sync writes - it's usually network clients writing to an NFS/CIFS share who risk issues.
 
Back
Top