FreeNAS Performance Weirdness

tonyb

Limp Gawd
Joined
Dec 30, 2010
Messages
130
So I've got a FreeNAS system I built. 3 2TB WD hard drives, AMD X2 5000+ processor, 3GB of RAM, PCIe Gigabit card.

The file system is ZFS, in a RAID5ish array, with encryption.

My write operations over NFS and SMB are sporadic. I get about 35 MB/s average, but looking at the network graph, it spikes to really high lvesl, then drops off.

Reads are actually slower, around 20 MB/s, when I copy data off.

Image-0018.png


This is measured from my Windows 7 workstation. The left part of the graph represents copying a single 4 GB file from my Workstation to the FreeNAS server. It spikes to about 700 Mb/s, which is about 87 MB/s.


The right part of the graph represents copying that same file back. That is more consistent, about 140 Mb/s, or 17 MB/s.

I know encrypted RAID5 is going to slow down writes (and even reads with encryption) but this doesn't seem quite right.
 
what version of FreeNAS/ZFS?

I had problems with FreeNAS 7 and NFS - which ultimately led me to Nexenta. With your specs, I don't think the overall throughput numbers are that far off for an encrypted RAIDZ array. What does your server CPU/Memory look like during read/write operations?
 
You may have to tune your transaction groups, but i don't see any local benchmarks. You should do local benchmarks first, see how fast you can write 20GB to your ZFS pool.

If the local benchmarks are high enough, then use the tuning variables like:

vfs.zfs.txg.write_limit_override: 100000000
vfs.zfs.txg.synctime: 2

So max 2 seconds or 100MB buffer, which ever comes first. Default is every 5 seconds with no explicit buffer size.
 
FreeBSD especially older FreeBSD would need tuning for decent ZFS performance. FreeNAS is based on the older FreeBSD 7.x release branch, where version 8.0 has the first production-ready ZFS release so FreeNAS used an alpha/beta quality ZFS implementation for quite some time. Now with 7.3 release the ZFS code from 8.0 was backported, but still this is a quite old implementation.

Regardless, try the tuning variables i supplied, that may solve your issue about stalling network transfers as FreeBSD by default uses too big transaction groups for your hardware to handle.
 
Thanks for the turning, I tried, and it smoothed things out a bit, but it still writes were faster than read (weird). And even a simple file copy will ruin file latency, causing jumpiness in watching a video stored on the same volume. I was using it as an NFS datastore for ESXi, but went to local disk because it was unusable.

I also checked for Ethernet errors in case there was some network weirdness going on. Both sides had clean error counters.
 
Like I said, I really really wanted to use FreeNAS for the exact same system - NFS datastore for ESXi. After a lot of head-pounding, I went with Nexenta and couldn't be happier. I recommend checking it out.
 
I've rebuilt the system. At first I tried FreeNAS .8 RC2, but the ZFS support there is bare-bones at best. I couldn't get an array built, and there's no encryption in the GUI at all for FreeNAS 8.

I went back to .72, and built a standard RAID 5 array. Even with the rebuild I/O, I'm getting 10 MB/s writes.

The parity of the entire array (3 x 2 TB drives) is going to take about 2 or 3 days. Less if I stopped copying stuff onto it, but even idle it will take a while.
 
I tried OpenFiler as well, but it won't install onto a flash drive without significant work. I don't want to have a spindle handle the boot drive, so FreeNAS it is.
 
Back
Top