Bonnie++ benchmarks with various ZFS configs

M

MrLie

Weaksauce
Joined
Sep 5, 2010
Messages
80
I just recently decided to perform a disk-upgrade to my ZFS, and decided this was the perfect time to play around abit to see what the difference in performance were with various ZFS configs, on the same hardware.
I also tried Sub.Mesa's LiveCD 0.1.5, but it wouldn't recognize my two controllers and see my Hitatchi-drives. Havnt had the time (or knowledge yet) to figure how to fix that.

At the moment I'm still playing around with the hardware, and are free to create and destroy pools at will. This is (supposed) to replace a WHS installation I have, and further down I intend to combine the two Norco's with a HP SAS Expander (allready got) and a HBA (need to buy) - probably not until next year. This is for home use only and will be limited by gbit network.


My hardware is as following:
SuperMicro C2SBX
AOC-SAT2-MV8 x2
Hitachi Deskstar 7K2000 HDS722020ALA330 32MB 2TB x16
8 GB ram
Intel E8400
OpenSolaris 2009.06


The Bonnie++ command I used on all tests were:
Code: 
bonnie++ -d /pool/bonnie -s 64000 -r 8000 -u root:root


2x raidz-2 on separate controllers:
Code: 
Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
opensolaris  64000M 109443  92 209021  29 140410  22 91601  91 542880  39  1420   3
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ 32550  99 +++++ +++ +++++ +++
opensolaris,64000M,109443,92,209021,29,140410,22,91601,91,542880,39,1420.5,3,16,+++++,+++,+++++,+++,+++++,+++,32550,99,+++++,+++,+++++,+++

2x raidz on separate controllers:
Code: 
Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
opensolaris  64000M 112338  93 393981  55 221649  36 92682  93 617735  46  1449   2
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ 31538  99 +++++ +++ +++++ +++
opensolaris,64000M,112338,93,393981,55,221649,36,92682,93,617735,46,1449.3,2,16,+++++,+++,+++++,+++,+++++,+++,31538,99,+++++,+++,+++++,+++

"raid 0", 8 drives, 1 controller:
Code: 
Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
opensolaris  64000M 115820  97 457238  58 267593  41 97249  96 777630  47  2173   3
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
opensolaris,64000M,115820,97,457238,58,267593,41,97249,96,777630,47,2172.9,3,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

"raid 0", 16 drives, 2 controllers:
Code: 
Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
opensolaris  64000M 116397  96 475891  60 270569  41 97368  96 761250  45  4029   5
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ 32621  99 +++++ +++ +++++ +++
opensolaris,64000M,116397,96,475891,60,270569,41,97368,96,761250,45,4028.8,5,16,+++++,+++,+++++,+++,+++++,+++,32621,99,+++++,+++,+++++,+++

raid10, mirror 16 drives, 1 drive from each controller in each pair:
Code: 
Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
opensolaris  64000M 115641  97 264596  34 186446  29 97551  96 779287  47  3345   4
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ 32615  99 +++++ +++ +++++ +++
opensolaris,64000M,115641,97,264596,34,186446,29,97551,96,779287,47,3344.6,4,16,+++++,+++,+++++,+++,+++++,+++,32615,99,+++++,+++,+++++,+++


Raid10, mirror 16 drives, all pairs on same controller:
Code: 
Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
opensolaris  64000M 115075  97 263799  34 186572  29 97470  96 777276  47  3250   4
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ 31881  99 +++++ +++ +++++ +++
opensolaris,64000M,115075,97,263799,34,186572,29,97470,96,777276,47,3249.7,4,16,+++++,+++,+++++,+++,+++++,+++,31881,99,+++++,+++,+++++,+++
 
Hey there!

I searched a bit why your controller wouldn't be seen using my distro, and i think i found the curlpit. You need the following setting in /boot/loader.conf:
hw.hptrr.attach_generic=0

You can also do this with livecd; in the boot menu that counts down towards 0 choose "Escape to loader prompt". Then enter "set hw.hptrr.attach_generic=0" (without the " "). After you pressed enter, simply enter "boot" + enter to continue booting.

This will prevent some Highpoint cards from trying to serve/handle your controller, while it should be handled by ata driver instead.

If you like to try my distro again, perhaps you should wait until tomorrow when i release my new 0.1.7-preview2.iso, with advanced benchmarking support and ZFS version 15.
 
That command worked like a charm - got 16 new disks to play with using your zfs-gui :)
I'll pay attention tomorrow for the 0.1.7 release, and start to mess around.
 
Yes, keep in mind that if you upgrade the Web-GUI, you still got older system version; so you need to redownload the 0.1.7-preview2 iso when it is available and re-install. I'll finish it tonight and post URL in my thread tomorrow. I'll include this variable so it should work out-of-the-box for you now.

Cheers!
 
Just did a upgrade through the web-gui and got this error message:

ZFSGURU-benchmark, version 1
Test size: 64.000 gigabytes (GiB)
Test rounds: 3
Cooldown period: 5 seconds
Number of disks: 1 disks
disk 1: gpt/(null)

* Test Settings: TS64; SC1;
* Tuning: none
* Stopping background processes: sendmail, moused, syslogd and cron
* Stopping Samba service

Secure Erase. Syntax error: "(" unexpected

* ERROR during "secure_erase"; got return value 2
cannot open 'gurubenchmarkpool': no such pool
Click to expand...

Guess the topic has moved pretty far towards your web-gui now - I'll continue the discussion about your gui there, and let this thread live (or die) with the bonnie++ tests I posted :)
 
Code: 
ruler@backup3:/# prtdiag
System Configuration: Supermicro X7DW3
BIOS Configuration: Phoenix Technologies LTD  1.2 11/04/2008
BMC Configuration: IPMI 1.0 (unknown)

==== Processor Sockets ====================================

Version                          Location Tag
-------------------------------- --------------------------
Intel(R) Xeon(R) CPU           L5420  @ 2.50GHz LGA771/CPU1
Intel(R) Xeon(R) CPU           L5420  @ 2.50GHz LGA771/CPU2

==== Memory Device Sockets ================================

Type        Status Set Device Locator      Bank Locator
----------- ------ --- ------------------- ----------------
DDR2 FBDIMM in use 1   DIMM1A              Bank1
DDR2 FBDIMM in use 1   DIMM1B              Bank1
DDR2 FBDIMM empty  1   DIMM1C              Bank1
DDR2 FBDIMM empty  1   DIMM1D              Bank1
DDR2 FBDIMM in use 1   DIMM2A              Bank2
DDR2 FBDIMM in use 1   DIMM2B              Bank2
DDR2 FBDIMM empty  1   DIMM2C              Bank2
DDR2 FBDIMM empty  1   DIMM2D              Bank2
DDR2 FBDIMM in use 1   DIMM3A              Bank3
DDR2 FBDIMM in use 1   DIMM3B              Bank3
DDR2 FBDIMM empty  1   DIMM3C              Bank3
DDR2 FBDIMM empty  1   DIMM3D              Bank3
DDR2 FBDIMM in use 1   DIMM4A              Bank4
DDR2 FBDIMM in use 1   DIMM4B              Bank4
DDR2 FBDIMM empty  1   DIMM4C              Bank4
DDR2 FBDIMM empty  1   DIMM4D              Bank4

==== On-Board Devices =====================================
ATI ES1000

==== Upgradeable Slots ====================================

ID  Status    Type             Description
--- --------- ---------------- ----------------------------
1   in use    PCI-X            PCIX#1-133MHz
2   in use    PCI-X            PCIX#2-133MHz
3   available PCI Express      PCIE#3-x8
4   available PCI Express      PCIE#4-x4
5   available PCI Express      PCIE#5-x8
6   available PCI Express      PCIE#6-x8

(2 SUPERMICRO AOC-SAT2-MV8 64-bit PCI-X133MHz SATA Controller Card)

  pool: tank12
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank12      ONLINE       0     0     0
          raidz3-0  ONLINE       0     0     0
            c6t3d0  ONLINE       0     0     0
            c6t2d0  ONLINE       0     0     0
            c6t1d0  ONLINE       0     0     0
            c6t0d0  ONLINE       0     0     0
            c5t7d0  ONLINE       0     0     0
            c5t6d0  ONLINE       0     0     0
            c5t5d0  ONLINE       0     0     0
            c5t4d0  ONLINE       0     0     0
            c5t3d0  ONLINE       0     0     0
            c5t2d0  ONLINE       0     0     0
            c5t1d0  ONLINE       0     0     0
            c5t0d0  ONLINE       0     0     0
            c4t7d0  ONLINE       0     0     0
            c4t6d0  ONLINE       0     0     0
            c4t5d0  ONLINE       0     0     0
            c4t4d0  ONLINE       0     0     0
            c4t3d0  ONLINE       0     0     0
            c4t2d0  ONLINE       0     0     0
            c4t1d0  ONLINE       0     0     0
            c4t0d0  ONLINE       0     0     0
        logs
          c6t5d0    ONLINE       0     0     0


(Intel SLC SSD)
(20 drives all seagate 1.5TB some 5900 some 7200)

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
backup3      64000M 99959  95 340771  58 296445  66 84116  99 984354  95 +++++ +++
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 28789  99 +++++ +++ +++++ +++ 25184 100 +++++ +++ +++++ +++
backup3,64000M,99959,95,340771,58,296445,66,84116,99,984354,95,+++++,+++,16,28789,99,+++++,+++,+++++,+++,25184,100,+++++,+++,+++++,+++
 
You must log in or register to reply here.
Back
Top