ZFS Build recommended Testing/Benchmarks

[Retsam_Evals]

Case - Supermicro SC846E16-R1200B
Backplane - BPN-SAS2-846EL1
Motherboard - SuperMicro X9SRL-F
CPU - E5-1620 v2 3.7GHz 4c/8t
RAM - 6x8 GB sticks (8x8 planned and I can borrow some for testing if dual channel will be a bottle neck)
HBA - SAS9207-8i
Drives - currently 14 x 3 TB WD Green (Intellipark set to 5 min)

I also have a couple of Mellanox ConnectX MNPH29B-XTC NICs if any one is interested in some 10 gb/s testing.

I have never setup ZFS before but would like to figure out the best setup before committing data to the array and being stuck with it. I have been debating between 2 x 11 drive raidz3 or 4 x 6 drive raidz2 with 4 TB or larger drives. I decided that since I'm in not hurry, I may as well test different setups.

So what benchmarks are recommended?
What implementations of ZFS should I look at?
What configs would you all like to see numbers on?
Is there a script to test a bunch of configs instead on me testing manually?
Any recommended advice?

Does any ZFS version support hot spares yet?

 

[_Gea]

So what benchmarks are recommended?
What implementations of ZFS should I look at?
What configs would you all like to see numbers on?
Is there a script to test a bunch of configs instead on me testing manually?
Any recommended advice?

Does any ZFS version support hot spares yet?

Your OS options

BSD
- FreeNAS
- NAS4Free
- ZFSGuru

Linux
- any distribution with additional ZoL support (not part of the OS)

Solaris and free forks based on Illumos
- Oracle Solaris

- NexentaStor
- OpenIndiana
- OmniOS
- SmartOS (KVM server use only, not for a NAS)
- some others

I prefer Solaris based solution especially OmniOS because ZFS was developped for Solaris with long profen stability for datapools and system pools with bootable snaps. Other features like Comstar (IB/FC/iSCSI), Crossbow (virtual networking) CIFS (Windows ACL compatible SMB server) are reasons.

Solaris supports IB hardware beside the newest generation and supports hotspares. There are versions with GUI like OpenIndiana or Solaris and serverbased CLI versions like OmniOS. For Solaris based options beside the commercial NexentaStor I wrote a web-UI with IB support and a ready to use appliance installer or a ready to use VM for a virtual NAS under ESXi.

For local tests you can use tools like bonnie, for remore tests tools like NAStester or tools like Crystal for benchmarks on blockbased targets.

 

[danswartz]

I just acquired a couple of connectx-2 boards. I linked vsphere host to host running omnios. Installed opensm on vsphere so srp will work. vsphere sees the target and creates datastore just fine. I would like to also be able to export the same zvol via iscsi for the benefit of other vsphere host without infiniband. I can't seem to see how to do this. vsphere can handle (AFAIK) multiple targets using same device, since it sees the vmfs signature on the device. But I can't seem to set up an iscsi target that also exports the same zvol (omnios tells me it is in use.) Any thoughts? Or am I out of luck?

 

[Olga-SAN]

intel i/o meter is classic scriptable and you can find a lot of ics files to match your workload

http://www.iometer.org

vdbench is king you can do pretty much everything with it

http://www.oracle.com/technetwork/server-storage/vdbench-downloads-1901681.html

jetstress if you need to simulate exchange

http://www.microsoft.com/en-us/download/details.aspx?id=36849

sqlio if you need sqlserver

http://www.microsoft.com/en-us/download/details.aspx?id=20163

microsoft pushes diskspd but i see no much point in it

http://blogs.technet.com/b/josebda/...for-both-local-disks-and-smb-file-shares.aspx

for fast & dirty numbers you can use atto disk benchmark

http://www.attotech.com/disk-benchmark/

Case - Supermicro SC846E16-R1200B
Backplane - BPN-SAS2-846EL1
Motherboard - SuperMicro X9SRL-F
CPU - E5-1620 v2 3.7GHz 4c/8t
RAM - 6x8 GB sticks (8x8 planned and I can borrow some for testing if dual channel will be a bottle neck)
HBA - SAS9207-8i
Drives - currently 14 x 3 TB WD Green (Intellipark set to 5 min)

I also have a couple of Mellanox ConnectX MNPH29B-XTC NICs if any one is interested in some 10 gb/s testing.

I have never setup ZFS before but would like to figure out the best setup before committing data to the array and being stuck with it. I have been debating between 2 x 11 drive raidz3 or 4 x 6 drive raidz2 with 4 TB or larger drives. I decided that since I'm in not hurry, I may as well test different setups.

So what benchmarks are recommended?
What implementations of ZFS should I look at?
What configs would you all like to see numbers on?
Is there a script to test a bunch of configs instead on me testing manually?
Any recommended advice?

Does any ZFS version support hot spares yet?

 

[kenned3]

Case - Supermicro SC846E16-R1200

Does any ZFS version support hot spares yet?

This works fine under Solaris and will auto online the spare when needed. I may be incorrect here, but I dont think this works under the other options (free as, zfs on Linux...)

 

[Aesma]

You didn't say what you intended to do with the server.

 

[cantalup]

This works fine under Solaris and will auto online the spare when needed. I may be incorrect here, but I dont think this works under the other options (free as, zfs on Linux...)

well not fully works on ZoL,
autoreplace is handled bu Zed ( must be running in the background) vi sh script.

if you pull one Drive, ZoL can Not do Autoreplace
Zol Can do do autoreplace on this condition:IO or checksum errors.


take a look on theri github -> https://github.com/zfsonlinux/zfs/blob/master/cmd/zed/zed.d/io-spare.sh#L03-22

Replace a device with a hot spare in response to IO or checksum errors.
# The following actions will be performed automatically when the number
# of errors exceed the limit set by ZED_SPARE_ON_IO_ERRORS or
# ZED_SPARE_ON_CHECKSUM_ERRORS.
#
# 1) FAULT the device on IO errors, no futher IO will be attempted.
# DEGRADE the device on checksum errors, the device is still
# functional and can be used to service IO requests.
# 2) Set the SES fault beacon for the device.
# 3) Replace the device with a hot spare if any are available.
#
# Once the hot sparing operation is complete either the failed device or
# the hot spare must be manually retired using the 'zpool detach' command.
# The 'autoreplace' functionality which would normally take care of this
# under Illumos has not yet been implemented.
#
# Full support for autoreplace is planned, but it requires that the full
# ZFS Diagnosis Engine be ported. In the meanwhile this script provides
# the majority of the expected hot spare functionality.

 

[Retsam_Evals]

Sorry about the lack of replies, but I have not been able to start testing due to the fact my case did not come with the front panel adapter.

I'm interested is the Solaris based OS's but from my research it seems the FreeBSD OS's have better hardware support and are more stable.

How does the napp-it web UI compare to freenas?
Is the any advantage to running FreeBSD vs freenas if I plan only to store data?

 

[/dev/null]

My FreeBSD box (FreeBSD 10) has been rock solid for several years. Use straight freebsd & you should be golden.

 

[sachem87184]

Sorry about the lack of replies, but I have not been able to start testing due to the fact my case did not come with the front panel adapter.

I'm interested is the Solaris based OS's but from my research it seems the FreeBSD OS's have better hardware support and are more stable.

How does the napp-it web UI compare to freenas?
Is the any advantage to running FreeBSD vs freenas if I plan only to store data?

That connector for the front panel is compatible with Supermicro motherboards. Correct me if I'm wrong but the JF1 header on the motherboard should directly connect to the chassis. I have several Supermicro chasses and this hasn't been a problem yet. It can only go on two ways so try both and see.

 

[cantalup]

Sorry about the lack of replies, but I have not been able to start testing due to the fact my case did not come with the front panel adapter.

I'm interested is the Solaris based OS's but from my research it seems the FreeBSD OS's have better hardware support and are more stable.

How does the napp-it web UI compare to freenas?
Is the any advantage to running FreeBSD vs freenas if I plan only to store data?

if you have time to learn in command line, this will save you a million time when dealing with problems
I always start with commad line and move to GUI, when knowing" how the way works".

one thing to remember, build a backup plan when your primary NAS is failing apart or in comma . safety first with a backup ...

 

[cantalup]

That connector for the front panel is compatible with Supermicro motherboards. Correct me if I'm wrong but the JF1 header on the motherboard should directly connect to the chassis. I have several Supermicro chasses and this hasn't been a problem yet. It can only go on two ways so try both and see.

ditto.
front panel SM connector always plug and forget for SM motherboards as I understand

if not SM motherboard, need a bit time using breadboard jumber cables to connect to motherboard