ZFS build newbie

Zedicus

Zedicus

[H]ard|Gawd
Joined
Nov 2, 2010
Messages
1,337
hardware 2x amd 6128HE
all of the vt-d and virtual stuff is enabled and working
48gb triple channel ddr3 i think it is 1333 speed
host has a adaptec 5405 with 2x10000rpm drives for host os (proxmox, overkill) and 2x500gb seagates for VM OS's)

all intel nics, i have 3 and only have 2 in use, 1 for host and one dedicated to the single vm that is running freenas

5x st3000dm001 on perc310 flashed IT PCI pass through
Raidz ZFS, tried with lz4 and with no compression maybe 1-3 MBs change
network does not seem to be the issue as i can open remote desktops and generate other traffic and it all seems fine.
freenas VM has access to 6 cores and 20gb ram

all gige network

getting about 33MBs write and read even on a local write say from host to ZFS
getting about 33MBs even if i do a copy from the raidz back to the raidz

if i do more then one read from the zfs the speed on each read halfs. so i get 2x reads at about 17MBs

i have a real raid5 array on a different server and i can pull and push 5x the traffic to it with out any issue. what am i doing wrong?
 
the ZFS is on the perc310 in IT mode, that is a HBA

the adaptec only houses the OS. the real raid stuff is actually fast, most of my server experience is relian on hardware raid so i am asking for ZFS help. as my HBA with ZFS in a RAIDZ is dirt slow.
 
Try breaking the array and test throughput to individual drives, see what you get that way to start with.
 
Did you test the performance locally or just over network? Maybe it is just the network that is slow?
 
Dalfo002 said:
Try breaking the array and test throughput to individual drives, see what you get that way to start with.
Click to expand...
i did this and individual drive writes are about 70 MB/s avg.


diehard said:
What protocol are you using? NFS, iSCSI, CIFS?
Click to expand...
when doing the copy over the network i was using CIFS.


omniscence said:
Did you test the performance locally or just over network? Maybe it is just the network that is slow?
Click to expand...
i did local and network testing. network does not seem to be the issue. its an intel nic and the same style that i have used before. other systems on the same network do not have issues.
 
What is the pool/dataset sync property set to? Is it still set sync=always?

You'll never get more than a single drives IOPs from raid-z but the sequential should be much better.
 
it is set to sync = always i was told other settings are 'unsafe' ?

i am getting under half of the write speed of a single disc.
 
Either you misunderstood or whoever told you 'use sync=always' had no idea what they were talking about. The sync default of 'standard' will do sync writes when the client requests it. A properly coded client will only do so when it matters. ex: you open a file, write several MB of data, then close it. The data being written can be written in asynchronous mode as long as the client expects to sync it at the end. e.g. depends on the protocol requirements. If you are backing a vmware datastore via NFS, vmware does ALL NFS writes as synchronous, so your performance on writes will suck. Unless you get a good SSD for SLOG device. Note that sequential write perf will suck balls in sync=always, because ZFS will be using an on-pool ZIL, and every logical record will cause a write to the pool for the ZIL and then another for the actual data write (this is only an approximate explanation), but it explains (roughly) why it sucks. So anyway, almost never does local write need sync=always.
 
In short, VMWare datastores over iSCSI and NFS need sync=always.

Your CIFS based local network probably doesnt need it.

Write performance will be terrible with sync=always unless you have a low latency SSD, ala Intel Enterprise ones.
 
This is not true. As I said, vsphere forces sync where it needs it, so sync=standard is fine.
 
danswartz said:
This is not true. As I said, vsphere forces sync where it needs it, so sync=standard is fine.
Click to expand...

False. ESXi has no clue as to the value, importance, or other attributes of the data that your VM is attempting to write. Just about every write will be sync write.

Not forcing sync on an ZFS pool containing a ESXi datastore is "fine" as long as you are fine with losing data and possibly corrupt VM's in the event of a power loss.
 
Last edited:
Leaving aside the quibble about 'where it needs it' (and iSCSI, about which I can't comment), my statement about vsphere as an NFS client absolutely is true. It forces sync for every write. Period. So setting sync=always is pointless. If you claim otherwise, you are doing so out of ignorance.
 
danswartz said:
Leaving aside the quibble about 'where it needs it' (and iSCSI, about which I can't comment), my statement about vsphere as an NFS client absolutely is true. It forces sync for every write. Period. So setting sync=always is pointless. If you claim otherwise, you are doing so out of ignorance.
Click to expand...

That was in reference to NFS using sync writes moreso than the need to set sync=always if using NFS, It wont really make a difference as the writes will be synchronous anyway.

You seem to be suggesting that VM's know which writes need to be synced, which is false.

iSCSI absolutely needs sync=always, as by default no writes are sync. At least with istgt which has been the FreeBSD standard for a while, im honestly not sure if Kernel Target will handle iSCSI SYNCHRONIZE CACHE commands from vsphere.
 
Last edited:
I tried to correct that confusing statement. To be clear, I am not making any assertions whatsoever except wrt NFS client in esxi, where it makes no difference if you set sync=always or sync=standard. Thanks for the info about iSCSI - I was curious if sync cache was being sent or not - I'd rather not trust it if I did use iSCSI :) I saw a page about the different iscsi targets in linux (a matrix) showing features, bugs, questions, etc... Don't remember the link right now though...
 
my ZFS pool will have no VM datastore on it. it will strictly be file storage. it may get written to locally but 98% will be write/read over CIFS.

to sync, or not to sync. that is my question?
 
danswartz said:
I tried to correct that confusing statement. To be clear, I am not making any assertions whatsoever except wrt NFS client in esxi, where it makes no difference if you set sync=always or sync=standard. Thanks for the info about iSCSI - I was curious if sync cache was being sent or not - I'd rather not trust it if I did use iSCSI :) I saw a page about the different iscsi targets in linux (a matrix) showing features, bugs, questions, etc... Don't remember the link right now though...
Click to expand...

Here is a comparison for Linux targets http://linux-iscsi.org/wiki/Features

FreeBSD is/has switched to a new native kernel implementation .. the latest FreeNAS has an "experimental mode" that enables it, and the next version it will be the default. .. lol getting even more off topic
Zedicus said:
my ZFS pool will have no VM datastore on it. it will strictly be file storage. it may get written to locally but 98% will be write/read over CIFS.

to sync, or not to sync. that is my question?
Click to expand...

Definately use sync=standard. Don't use sync=disable as i believe there is some debate that this can actually disable ZFS metadata sync on the pool, which could be bad.
 
also how do i check to see if my pool is configured correctly for 4k drives? some distros say its set that way default, some do not.
 
Zedicus said:
also how do i check to see if my pool is configured correctly for 4k drives? some distros say its set that way default, some do not.
Click to expand...

It should be using SMART data to pull the info from the drives and configure it, the problem can be when drives emulate 512b. I believe a lot of distro's are now forcing 4k because its far better to have 512b drives on 4k than vice-versa.

Run a "zdb | grep ashft" in the console, 9 is 512b.. 12 is 4k.
 
sync set standard
ashift reports correct for 4k
my network is showing zilch for utilization, gig network, intel cards.
now i am not even getting 25MB/s purely sequencial writes, nothing else happening at all on the ZFS test drives. compression off.

i guess i should mention this has been on BSD based distros. and my config is not really solaris friendly. should i give ZoL a try or something?

also i am using a single dataset on my 5 drive raidz. would a different configuration work better?

its about the same doing local stuff on the ZFS also.

and same results with NFS


but like i said if i attach a drive directly, format it say as EXT4 and mount and share it then i get around 80MB/s which seems to be normal.
 
Last edited:
more testing i built a 2x2 mirrored pool, 4 drives total. thinking o.k. this has to be faster.

i get the EXACT same speeds as a 5 drive raidz is that even possible?
 
Most benches are sequential where you can expect the following:
a 5 disk Raid-Z reads/writes simulaniosly form 4 datadisks: read/write performance= 4 x a single disk
a raid-10 from 4 disks: reads simulaniously from 4 disks=same as raid-z, write performance=2 x a single disk

if you look at IOPS:
Raid-Z where all disks must be positioned for every read: 1 x a single disk (about 100 iops)
Mirror (2 vdevs)= 2 x a single disk (about 200 iops)
so yes, this is possible, on sequential write a raid-z is faster on random read/write the mirror is faster

Regarding performance
You may compare s Solaris based ZFS system like OmniOS. From many reports this can be faster with NFS/CIFS out of the box
 
my KVM virtual environment does not like solaris based kernels and apparently i have not gotten BSD systems down correctly either. i am getting 480MB/s writes with ZoL though so i may go that route. theres no all in one NAS based on ZoL but i can use the napp-it gui for the ZFS stuff and and webmin for about everything else.

maybe there are some ways of making solaris or BSD work correctly under KVM? they seem to perform fine until ZFS is used in KVM on one of those platforms.
 
There is at least one ZoL that provides a gui. osnexus.com. They do have a free version with limitations.
 
osnexus looks nice but their 'free' version is up to 10tb.

i am adequate enough with linux that i do not need a gui, so the napp-it utility for som ZoL config and the rest of the stuff i can handle.

so far ZoL on a KVM VM with PCI-passthrough has performed well. i hope to test more and put it into use soon.
 
You must log in or register to reply here.
Back
Top