• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Performance Tests, thoughts?

J

JayG30

n00b
Joined
Aug 6, 2008
Messages
28
Hello everyone. I've done some testing on my new ZFS build.
Since I'm not as well versed in ZFS as many others here, I'd like to get some thoughts.

System Hardware Specs
1 x Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz
1 x ASRock Z87E-ITX
2 x 8GB Crucial Ballistix Sport DDR3-1600
6 x Seagate 4TB NAS HDD (ST4000VN000)
1 x SILVERSTONE ST55F-G 550W
1 x Fractal Design Node 304
1 x Cooler Master Hyper 212 EVO
Using the onboard Intel I217V NIC and 6 SATAIII ports.

System Software Specs
FreeNAS-9.1.0-BETA-a83fe91-x64
1 x RaidZ2

Network
Tests were either performed directly on the server via SSH or from a Windows 7 machine connected to the same 1Gbps router as the server. Tests done with Iperf appear to show expected speeds.


Testing
I did some diskinfo -t that I won't bother posting unless someone wants them. I didn't see anything wrong.
Ran SMART short test, no issues their either.

CrystalDiskMark
ospl.png

Code: 
-----------------------------------------------------------------------
CrystalDiskMark 3.0.2 x64 (C) 2007-2013 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :   109.845 MB/s
          Sequential Write :   100.679 MB/s
         Random Read 512KB :   105.917 MB/s
        Random Write 512KB :    97.526 MB/s
    Random Read 4KB (QD=1) :    10.451 MB/s [  2551.5 IOPS]
   Random Write 4KB (QD=1) :     8.888 MB/s [  2170.0 IOPS]
   Random Read 4KB (QD=32) :   111.510 MB/s [ 27224.1 IOPS]
  Random Write 4KB (QD=32) :    25.104 MB/s [  6128.9 IOPS]

  Test : 1000 MB [Z: 0.1% (20.5/14601.7 GB)] (x5)
  Date : 2013/07/07 23:11:32
    OS : Windows 7 Ultimate Edition SP1 [6.1 Build 7601] (x64)

ATTO
kbng.png


Write/Read of 30GB file.
If I'm understanding correctly, when you write/read files smaller then your RAM you will get extremely fast speeds due to caching.
To see what real disk performance is like we should test with writes/reads of data above that amount?
Code: 
[root@freenas] /mnt/store# dd if=/dev/zero of=/mnt/store/zerofile.000 bs=10M count=3200
3200+0 records in
3200+0 records out
33554432000 bytes transferred in 62.317696 secs (538441472 bytes/sec)

[root@freenas] /mnt/store# dd if=/mnt/store/zerofile.000 of=/dev/zero bs=10M count=3200
3200+0 records in
3200+0 records out
33554432000 bytes transferred in 63.747565 secs (526364136 bytes/sec)

IOZONE (pastbin)
Code: 
	Iozone: Performance Test of File I/O
	        Version $Revision: 3.397 $
		Compiled for 64 bit mode.
		Build: freebsd 

	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
	             Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.
	             Ben England.

	Run began: Sat Jul  6 21:42:49 2013


	Machine = FreeBSD freenas.local 9.1-STABLE FreeBSD 9.1-STABLE #0 r+7f710c8: Tu	Include fsync in write timing
	Include fsync in write timing
	CPU utilization Resolution = 0.000 seconds.
	CPU utilization Excel chart enabled
	Record Size 128 KB
	File size set to 40960 KB
	Percent read in mix test is 70
	Command line used: iozone -M -e -+u -T -t 32 -r 128k -s 40960 -i 0 -i 1 -i 2 -i 8 -+p 70 -C
	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
	Throughput test with 32 threads
	Each thread writes a 40960 Kbyte file in 128 Kbyte records

	Children see throughput for 32 initial writers 	=  488913.72 KB/sec
	Parent sees throughput for 32 initial writers 	=  172126.88 KB/sec
	Min throughput per thread 			=    7703.42 KB/sec 
	Max throughput per thread 			=   59566.02 KB/sec
	Avg throughput per thread 			=   15278.55 KB/sec
	Min xfer 					=    8064.00 KB
	CPU Utilization: Wall time    2.687    CPU time    9.738    CPU utilization 362.35 %

	Child[0] xfer count =    4352.00 KB, Throughput =    4009.30 KB/sec, wall= 2.687, cpu= 0.372, %= 13.84
	Child[1] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.660, cpu= 0.253, %=  9.50
	Child[2] xfer count =   25344.00 KB, Throughput =   24240.02 KB/sec, wall= 2.686, cpu= 0.352, %= 13.12
	Child[3] xfer count =    4352.00 KB, Throughput =    4160.65 KB/sec, wall= 2.687, cpu= 0.372, %= 13.83
	Child[4] xfer count =   32896.00 KB, Throughput =   31312.02 KB/sec, wall= 2.679, cpu= 0.331, %= 12.35
	Child[5] xfer count =    4352.00 KB, Throughput =    4110.85 KB/sec, wall= 2.687, cpu= 0.372, %= 13.84
	Child[6] xfer count =    8192.00 KB, Throughput =    7886.55 KB/sec, wall= 2.686, cpu= 0.353, %= 13.12
	Child[7] xfer count =    8192.00 KB, Throughput =    7785.44 KB/sec, wall= 2.686, cpu= 0.353, %= 13.12
	Child[8] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.660, cpu= 0.252, %=  9.49
	Child[9] xfer count =   29312.00 KB, Throughput =   27995.29 KB/sec, wall= 2.686, cpu= 0.352, %= 13.12
	Child[10] xfer count =    4224.00 KB, Throughput =    3989.86 KB/sec, wall= 2.687, cpu= 0.372, %= 13.84
	Child[11] xfer count =   33792.00 KB, Throughput =   32310.91 KB/sec, wall= 2.684, cpu= 0.348, %= 12.95
	Child[12] xfer count =   12544.00 KB, Throughput =   12099.72 KB/sec, wall= 2.684, cpu= 0.348, %= 12.97
	Child[13] xfer count =   37888.00 KB, Throughput =   36161.47 KB/sec, wall= 2.676, cpu= 0.318, %= 11.87
	Child[14] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.660, cpu= 0.252, %=  9.49
	Child[15] xfer count =   12416.00 KB, Throughput =   11724.95 KB/sec, wall= 2.680, cpu= 0.339, %= 12.65
	Child[16] xfer count =   16640.00 KB, Throughput =   15981.28 KB/sec, wall= 2.675, cpu= 0.316, %= 11.80
	Child[17] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.641, cpu= 0.245, %=  9.26
	Child[18] xfer count =   33280.00 KB, Throughput =   32310.78 KB/sec, wall= 2.671, cpu= 0.298, %= 11.17
	Child[19] xfer count =   15488.00 KB, Throughput =   15334.30 KB/sec, wall= 2.663, cpu= 0.271, %= 10.16
	Child[20] xfer count =    3968.00 KB, Throughput =    3857.63 KB/sec, wall= 2.670, cpu= 0.294, %= 11.02
	Child[21] xfer count =    8064.00 KB, Throughput =    7819.04 KB/sec, wall= 2.677, cpu= 0.326, %= 12.17
	Child[22] xfer count =    8192.00 KB, Throughput =    7950.84 KB/sec, wall= 2.667, cpu= 0.286, %= 10.72
	Child[23] xfer count =   40960.00 KB, Throughput =   39844.32 KB/sec, wall= 1.028, cpu= 0.086, %=  8.35
	Child[24] xfer count =   23936.00 KB, Throughput =   23092.88 KB/sec, wall= 2.665, cpu= 0.278, %= 10.44
	Child[25] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.641, cpu= 0.245, %=  9.27
	Child[26] xfer count =   37504.00 KB, Throughput =   35693.14 KB/sec, wall= 2.679, cpu= 0.331, %= 12.36
	Child[27] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.659, cpu= 0.252, %=  9.48
	Child[28] xfer count =   15360.00 KB, Throughput =   59566.02 KB/sec, wall= 1.104, cpu= 0.252, %= 22.82
	Child[29] xfer count =    8192.00 KB, Throughput =    7703.42 KB/sec, wall= 2.665, cpu= 0.278, %= 10.45
	Child[30] xfer count =    8064.00 KB, Throughput =    7756.73 KB/sec, wall= 2.681, cpu= 0.344, %= 12.83
	Child[31] xfer count =   25088.00 KB, Throughput =   24216.30 KB/sec, wall= 2.671, cpu= 0.298, %= 11.16

	Children see throughput for 32 rewriters 	=  754212.42 KB/sec
	Parent sees throughput for 32 rewriters 	=  523181.80 KB/sec
	Min throughput per thread 			=   16350.46 KB/sec 
	Max throughput per thread 			=  239399.38 KB/sec
	Avg throughput per thread 			=   23569.14 KB/sec
	Min xfer 					=   40960.00 KB
	CPU utilization: Wall time    2.505    CPU time    7.190    CPU utilization 287.00 %

	Child[0] xfer count =   40960.00 KB, Throughput =   16351.31 KB/sec, wall= 2.505, cpu= 0.326, %= 13.00
	Child[1] xfer count =   40960.00 KB, Throughput =   16849.61 KB/sec, wall= 2.431, cpu= 0.109, %=  4.47
	Child[2] xfer count =   40960.00 KB, Throughput =   16863.51 KB/sec, wall= 2.429, cpu= 0.095, %=  3.93
	Child[3] xfer count =   40960.00 KB, Throughput =   16479.75 KB/sec, wall= 2.485, cpu= 0.286, %= 11.51
	Child[4] xfer count =   40960.00 KB, Throughput =  239399.38 KB/sec, wall= 0.298, cpu= 0.298, %=100.00
	Child[5] xfer count =   40960.00 KB, Throughput =   16351.37 KB/sec, wall= 2.505, cpu= 0.326, %= 13.00
	Child[6] xfer count =   40960.00 KB, Throughput =   16350.46 KB/sec, wall= 2.505, cpu= 0.326, %= 13.02
	Child[7] xfer count =   40960.00 KB, Throughput =   16377.42 KB/sec, wall= 2.501, cpu= 0.314, %= 12.57
	Child[8] xfer count =   40960.00 KB, Throughput =   16604.74 KB/sec, wall= 2.467, cpu= 0.225, %=  9.12
	Child[9] xfer count =   40960.00 KB, Throughput =   16541.75 KB/sec, wall= 2.476, cpu= 0.256, %= 10.35
	Child[10] xfer count =   40960.00 KB, Throughput =   16479.23 KB/sec, wall= 2.486, cpu= 0.286, %= 11.50
	Child[11] xfer count =   40960.00 KB, Throughput =   16535.55 KB/sec, wall= 2.477, cpu= 0.262, %= 10.58
	Child[12] xfer count =   40960.00 KB, Throughput =   16470.42 KB/sec, wall= 2.487, cpu= 0.288, %= 11.57
	Child[13] xfer count =   40960.00 KB, Throughput =   16479.49 KB/sec, wall= 2.486, cpu= 0.286, %= 11.51
	Child[14] xfer count =   40960.00 KB, Throughput =   16532.66 KB/sec, wall= 2.478, cpu= 0.264, %= 10.67
	Child[15] xfer count =   40960.00 KB, Throughput =   16541.44 KB/sec, wall= 2.476, cpu= 0.257, %= 10.38
	Child[16] xfer count =   40960.00 KB, Throughput =   16569.05 KB/sec, wall= 2.472, cpu= 0.245, %=  9.91
	Child[17] xfer count =   40960.00 KB, Throughput =   16541.87 KB/sec, wall= 2.476, cpu= 0.256, %= 10.35
	Child[18] xfer count =   40960.00 KB, Throughput =   16683.12 KB/sec, wall= 2.455, cpu= 0.189, %=  7.71
	Child[19] xfer count =   40960.00 KB, Throughput =   16620.32 KB/sec, wall= 2.464, cpu= 0.224, %=  9.08
	Child[20] xfer count =   40960.00 KB, Throughput =   16701.86 KB/sec, wall= 2.452, cpu= 0.177, %=  7.21
	Child[21] xfer count =   40960.00 KB, Throughput =   16683.49 KB/sec, wall= 2.455, cpu= 0.189, %=  7.71
	Child[22] xfer count =   40960.00 KB, Throughput =   16622.70 KB/sec, wall= 2.464, cpu= 0.224, %=  9.08
	Child[23] xfer count =   40960.00 KB, Throughput =   16700.32 KB/sec, wall= 2.453, cpu= 0.178, %=  7.24
	Child[24] xfer count =   40960.00 KB, Throughput =   16779.61 KB/sec, wall= 2.441, cpu= 0.145, %=  5.93
	Child[25] xfer count =   40960.00 KB, Throughput =   16738.12 KB/sec, wall= 2.447, cpu= 0.156, %=  6.38
	Child[26] xfer count =   40960.00 KB, Throughput =   16750.31 KB/sec, wall= 2.445, cpu= 0.153, %=  6.25
	Child[27] xfer count =   40960.00 KB, Throughput =   16680.00 KB/sec, wall= 2.456, cpu= 0.199, %=  8.09
	Child[28] xfer count =   40960.00 KB, Throughput =   16683.35 KB/sec, wall= 2.455, cpu= 0.189, %=  7.72
	Child[29] xfer count =   40960.00 KB, Throughput =   16822.26 KB/sec, wall= 2.435, cpu= 0.116, %=  4.78
	Child[30] xfer count =   40960.00 KB, Throughput =   16627.70 KB/sec, wall= 2.463, cpu= 0.215, %=  8.74
	Child[31] xfer count =   40960.00 KB, Throughput =   16800.26 KB/sec, wall= 2.438, cpu= 0.130, %=  5.35

	Children see throughput for 32 readers 		= 14485534.00 KB/sec
	Parent sees throughput for 32 readers 		= 14110598.57 KB/sec
	Min throughput per thread 			=       0.00 KB/sec 
	Max throughput per thread 			= 3676190.50 KB/sec
	Avg throughput per thread 			=  452672.94 KB/sec
	Min xfer 					=       0.00 KB
	CPU utilization: Wall time    0.045    CPU time    0.147    CPU utilization 326.44 %

	Child[0] xfer count =   39552.00 KB, Throughput = 3541561.00 KB/sec, wall= 0.034, cpu= 0.034, %=100.00
	Child[1] xfer count =   40832.00 KB, Throughput = 3659456.00 KB/sec, wall= 0.034, cpu= 0.034, %=100.00
	Child[2] xfer count =   40320.00 KB, Throughput = 3608326.50 KB/sec, wall= 0.045, cpu= 0.045, %=100.00
	Child[3] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[4] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[5] xfer count =   40960.00 KB, Throughput = 3676190.50 KB/sec, wall= 0.034, cpu= 0.034, %=100.00
	Child[6] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[7] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[8] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[9] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 85.44
	Child[10] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[11] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[12] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 91.18
	Child[13] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 43.39
	Child[14] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[15] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[16] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[17] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[18] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 99.34
	Child[19] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 79.89
	Child[20] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 50.33
	Child[21] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[22] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[23] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[24] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 49.34
	Child[25] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[26] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[27] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[28] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[29] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 79.89
	Child[30] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[31] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00

	Children see throughput for 32 re-readers 	= 14484478.75 KB/sec
	Parent sees throughput for 32 re-readers 	= 14098699.61 KB/sec
	Min throughput per thread 			=       0.00 KB/sec 
	Max throughput per thread 			= 3694055.00 KB/sec
	Avg throughput per thread 			=  452639.96 KB/sec
	Min xfer 					=       0.00 KB
	CPU utilization: Wall time    0.044    CPU time    0.141    CPU utilization 319.24 %

	Child[0] xfer count =   40064.00 KB, Throughput = 3607739.75 KB/sec, wall= 0.033, cpu= 0.033, %=100.00
	Child[1] xfer count =   40320.00 KB, Throughput = 3628455.25 KB/sec, wall= 0.044, cpu= 0.044, %=100.00
	Child[2] xfer count =   40960.00 KB, Throughput = 3694055.00 KB/sec, wall= 0.030, cpu= 0.030, %=100.00
	Child[3] xfer count =   39424.00 KB, Throughput = 3554228.75 KB/sec, wall= 0.033, cpu= 0.033, %=100.00
	Child[4] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[5] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[6] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[7] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 50.33
	Child[8] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 93.21
	Child[9] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 55.92
	Child[10] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[11] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 74.02
	Child[12] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[13] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[14] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[15] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[16] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 59.92
	Child[17] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 44.15
	Child[18] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[19] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[20] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[21] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 59.92
	Child[22] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[23] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 49.34
	Child[24] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 98.69
	Child[25] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[26] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 74.02
	Child[27] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 74.02
	Child[28] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[29] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 79.89
	Child[30] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[31] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00

	Children see throughput for 32 random readers 	= 14411108.50 KB/sec
	Parent sees throughput for 32 random readers 	= 13713301.84 KB/sec
	Min throughput per thread 			=       0.00 KB/sec 
	Max throughput per thread 			= 3679497.25 KB/sec
	Avg throughput per thread 			=  450347.14 KB/sec
	Min xfer 					=       0.00 KB
	CPU utilization: Wall time    0.045    CPU time    0.158    CPU utilization 352.35 %

	Child[0] xfer count =   40832.00 KB, Throughput = 3659065.00 KB/sec, wall= 0.045, cpu= 0.045, %=100.00
	Child[1] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[2] xfer count =   39680.00 KB, Throughput = 3556211.00 KB/sec, wall= 0.045, cpu= 0.045, %=100.00
	Child[3] xfer count =   39168.00 KB, Throughput = 3516335.25 KB/sec, wall= 0.034, cpu= 0.034, %=100.00
	Child[4] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[5] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 56.87
	Child[6] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[7] xfer count =   40960.00 KB, Throughput = 3679497.25 KB/sec, wall= 0.034, cpu= 0.034, %=100.00
	Child[8] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[9] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[10] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[11] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[12] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[13] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 54.37
	Child[14] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[15] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[16] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[17] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[18] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[19] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 79.89
	Child[20] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 69.91
	Child[21] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[22] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[23] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 66.23
	Child[24] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[25] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[26] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 82.06
	Child[27] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 83.89
	Child[28] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[29] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[30] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[31] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00

	Children see throughput for 32 mixed workload 	= 14513196.25 KB/sec
	Parent sees throughput for 32 mixed workload 	=  160698.87 KB/sec
	Min throughput per thread 			=       0.00 KB/sec 
	Max throughput per thread 			= 3670613.75 KB/sec
	Avg throughput per thread 			=  453537.38 KB/sec
	Min xfer 					=       0.00 KB
	CPU utilization: Wall time    0.998    CPU time    1.218    CPU utilization 122.12 %

	Child[0] xfer count =   40064.00 KB, Throughput = 3582588.00 KB/sec, wall= 0.039, cpu= 0.039, %=100.00
	Child[1] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.269, cpu= 0.053, %= 19.72
	Child[2] xfer count =   40576.00 KB, Throughput = 3629997.25 KB/sec, wall= 0.039, cpu= 0.039, %=100.00
	Child[3] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[4] xfer count =   40576.00 KB, Throughput = 3629997.25 KB/sec, wall= 0.044, cpu= 0.044, %=100.00
	Child[5] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[6] xfer count =   40960.00 KB, Throughput = 3670613.75 KB/sec, wall= 0.039, cpu= 0.039, %=100.00
	Child[7] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[8] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 42.65
	Child[9] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %= 42.65
	Child[10] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[11] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[12] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[13] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[14] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[15] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.035, cpu= 0.012, %= 33.98
	Child[16] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[17] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[18] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.269, cpu= 0.053, %= 19.72
	Child[19] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[20] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.000, cpu= 0.000, %=100.00
	Child[21] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.251, cpu= 0.014, %=  5.43
	Child[22] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.269, cpu= 0.053, %= 19.77
	Child[23] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.398, cpu= 0.092, %= 23.07
	Child[24] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.998, cpu= 0.098, %=  9.85
	Child[25] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.279, cpu= 0.092, %= 33.00
	Child[26] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.997, cpu= 0.098, %=  9.84
	Child[27] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.997, cpu= 0.098, %=  9.84
	Child[28] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.998, cpu= 0.098, %=  9.85
	Child[29] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.997, cpu= 0.098, %=  9.84
	Child[30] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.997, cpu= 0.098, %=  9.84
	Child[31] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 0.998, cpu= 0.099, %=  9.90

	Children see throughput for 32 random writers 	=  549492.20 KB/sec
	Parent sees throughput for 32 random writers 	=   58424.30 KB/sec
	Min throughput per thread 			=       0.00 KB/sec 
	Max throughput per thread 			=  280223.31 KB/sec
	Avg throughput per thread 			=   17171.63 KB/sec
	Min xfer 					=       0.00 KB
	CPU utilization: Wall time    2.716    CPU time    9.476    CPU utilization 348.88 %

	Child[0] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.427, cpu= 0.302, %= 12.45
	Child[1] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.425, cpu= 0.302, %= 12.46
	Child[2] xfer count =   39424.00 KB, Throughput =   89756.22 KB/sec, wall= 2.442, cpu= 0.338, %= 13.85
	Child[3] xfer count =   39808.00 KB, Throughput =   90630.07 KB/sec, wall= 0.507, cpu= 0.293, %= 57.77
	Child[4] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.432, cpu= 0.302, %= 12.42
	Child[5] xfer count =   39040.00 KB, Throughput =   88882.59 KB/sec, wall= 2.421, cpu= 0.335, %= 13.85
	Child[6] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.425, cpu= 0.302, %= 12.45
	Child[7] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.716, cpu= 0.309, %= 11.38
	Child[8] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.433, cpu= 0.309, %= 12.69
	Child[9] xfer count =   40960.00 KB, Throughput =  280223.31 KB/sec, wall= 0.146, cpu= 0.036, %= 24.90
	Child[10] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.412, cpu= 0.306, %= 12.68
	Child[11] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.708, cpu= 0.302, %= 11.16
	Child[12] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.427, cpu= 0.302, %= 12.45
	Child[13] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.411, cpu= 0.299, %= 12.41
	Child[14] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.430, cpu= 0.302, %= 12.43
	Child[15] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.425, cpu= 0.302, %= 12.46
	Child[16] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.427, cpu= 0.302, %= 12.45
	Child[17] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.425, cpu= 0.302, %= 12.45
	Child[18] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.433, cpu= 0.302, %= 12.43
	Child[19] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.427, cpu= 0.302, %= 12.44
	Child[20] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.425, cpu= 0.302, %= 12.45
	Child[21] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.432, cpu= 0.302, %= 12.42
	Child[22] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.425, cpu= 0.302, %= 12.46
	Child[23] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.432, cpu= 0.302, %= 12.42
	Child[24] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.425, cpu= 0.302, %= 12.45
	Child[25] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.427, cpu= 0.302, %= 12.45
	Child[26] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.425, cpu= 0.302, %= 12.46
	Child[27] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.433, cpu= 0.302, %= 12.43
	Child[28] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.427, cpu= 0.302, %= 12.44
	Child[29] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.425, cpu= 0.302, %= 12.45
	Child[30] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.425, cpu= 0.302, %= 12.45
	Child[31] xfer count =       0.00 KB, Throughput =       0.00 KB/sec, wall= 2.715, cpu= 0.302, %= 11.14

iozone test complete.

I can perform other tests if it helps anyone diagnose potential performance issues.

What I am experiencing in real world file/folder copies of CIFS shares is as follows. Copying 2GB worth of video (4x500MB) windows file transfer shows ~120MB/s and then starts to drop down to 90...80..70MB/s half way through. I'm not sure if that is the expected behavior. When I transfer anything really large I tend to see transfer speeds cut in half to about 60MB/s. I believe this has something to do with the RAM amount and caching, but not sure.

Also, it appears that if I set my drives to advanced power management "level 64" that I am experiencing delays/pauses and low transfer speeds. When I look at top I can see that when these issues occur RAM has been released (14GB free) and when the transfer is started it slowly attempts to start using more of it (speed increases but not to expected levels). When I disable advanced power management, I do not see these issues and free RAM shows 7.8GB instead.

A side note, even though I'm not going to bother with HDD standby, I noticed that if I didn't comment out certain logs in /etc/syslog.conf I could not get them to spin down. I didn't go a ton of investigating but I know I had commented out maillog and cron (cron alone didn't resolve it, so it might not have been my issue). Just something I figured I'd mention.
 
Last edited:
I did another iozone test.
iozone -a -I -b iozone.xls

From my understanding -I should avoid the buffer cache.
-I Use DIRECT IO if possible for all file operations. Tells the filesystem that all operations to the file are to bypass the buffer cache and go directly to disk.
Click to expand...

Results seem high, but maybe I'm interpreting the results incorrectly.
Anybody?

Download excel file since it won't format correctly in CODE.

s8Ww6Lr6


ESS4Mn0e


eNjOMnbd


7edPlm0c


Code: 
iozone -a -I -b iozone.xls 													
 													
The top row is records sizes, the left column is file sizes													
Writer Report													
	4	8	16	32	64	128	256	512	1024	2048	4096	8192	16384
64	278227	552557	998622	1679761	3363612								
128	228571	449337	795593	1471662	2621367	4557257							
256	354042	663292	1248819	2097948	3326279	5217259	5569035						
512	477982	853596	1442420	2189509	2860238	6174377	5951911	5886650					
1024	564844	984676	1544504	2174880	2760606	6519323	6441107	6559147	6393168				
2048	627057	1051295	1634681	2285891	2844532	6780783	6917293	6962145	6717153	6583305			
4096	669942	1120026	1691796	2299600	2805423	7264812	7185807	6703539	6651630	6060333	5681538		
8192	664832	1115945	1672836	2281249	2585809	7012469	6804170	6861236	6558696	7008178	5461897	5041185	
16384	699094	1157306	1717608	2320991	2788282	7034583	7062054	7083893	7067864	6646751	5336686	5274831	5115413
32768	0	0	0	0	2813212	7251258	7173670	7151646	7042443	7042443	5931812	5359538	5008926
65536	0	0	0	0	2799967	7246251	7060678	7059046	7080502	7015089	6582671	5272344	5091009
131072	0	0	0	0	2771605	7128501	7055586	7168657	7174926	6966005	6630849	5383703	4979545
262144	0	0	0	0	2812490	7226411	7151443	7058839	7226174	6935196	6836831	5559918	5153731
524288	0	0	0	0	2798007	7286943	7100728	7199294	7181239	3604762	7110440	5582158	3832347
Re-writer Report													
	4	8	16	32	64	128	256	512	1024	2048	4096	8192	16384
64	718114	1279447	2892445	4643754	5389653								
128	919605	1543594	3036502	5122535	6406138	7582312							
256	853662	1622442	2872459	5117791	5938650	7314033	7735574						
512	840566	1629590	2995907	4927617	5886650	8018812	7758090	7758090					
1024	869284	1499217	2829721	4898425	6244448	8988941	8542002	8474583	8391792				
2048	856611	1585208	2767549	4531020	6583305	8940345	8253158	8357547	8565901	8325147			
4096	854218	1598156	2801763	4757904	7047267	9729195	9119767	8677553	8077780	7515892	7001316		
8192	820260	1564237	2728737	4563770	6930432	9492149	8761203	8498983	8455064	8828739	6591409	6220301	
16384	847198	1585926	2813742	4797571	7010899	9416457	9167723	9001994	8963247	8392905	7556734	6481253	5946749
32768	0	0	0	0	7104695	9503581	9160883	9225449	9049892	8906790	7193946	6474477	5944898
65536	0	0	0	0	7205790	9654389	9156856	9166017	9017178	9013334	7826194	6631904	5845753
131072	0	0	0	0	7099870	9498067	9146772	9173481	9147989	8843095	8467838	6582261	5952907
262144	0	0	0	0	7194259	9587916	9237279	8250763	9210969	9125652	8918829	6696915	4496468
524288	0	0	0	0	7264507	6020139	6431626	5460877	5462762	7303279	9357434	6826936	6065621
Reader Report													
	4	8	16	32	64	128	256	512	1024	2048	4096	8192	16384
64	2133730	3958892	6421025	10821524	22737791								
128	2326073	3895854	6406138	10779307	15881078	18012359							
256	2266207	4197489	6554972	10245071	14164395	16072608	16072608						
512	2275345	3736016	5886650	10044089	12215097	16049269	16049269	17069844					
1024	2240689	4048778	6244448	9569770	13305116	15516181	15295157	16789959	17331995				
2048	2011785	3436240	5044577	6780783	12805399	14321195	13949098	15854271	16525279	16915790			
4096	1984550	3279348	4971306	6847827	9545402	10089154	10319444	12083134	12229356	11704457	10502389		
8192	1913146	3226628	4853203	6429825	8844647	9502650	9513174	10151170	10792033	11038189	8366536	8127085	
16384	1975900	3358098	5081746	6998761	8659420	9299226	9486653	9942333	10475840	10577425	9082902	7915343	7721458
32768	0	0	0	0	8742487	9338277	9558440	9990615	9391880	10519785	8350481	7822451	7510405
65536	0	0	0	0	8863367	2752445	9486790	9920952	10294742	10389293	9068344	7748320	7499096
131072	0	0	0	0	8816857	9392428	9495934	9887729	10138456	10056299	3420649	7597110	7361505
262144	0	0	0	0	8826044	9415416	9507661	9938386	10240035	10191061	9108567	7744959	7447459
524288	0	0	0	0	6049386	9516252	9717858	10075116	10303600	8382570	9168793	7728513	7522945
Re-reader Report													
	4	8	16	32	64	128	256	512	1024	2048	4096	8192	16384
64	2278628	3958892	7100397	12902017	15972885								
128	2377579	4267461	6727225	11720614	18637664	18637664							
256	2285501	4281170	6761355	10651598	15164624	17096249	15835569						
512	2297251	4163357	6246213	10044089	14240069	16543832	16049269	17069844					
1024	2281152	4178773	6479979	9855234	14044758	16280798	15080341	17124680	17331995				
2048	2250555	4120256	6503556	9318595	13836752	15854271	15289867	16493549	17360217	17360217			
4096	2181070	3931494	6300364	9614850	13007113	15059417	11601703	14177082	14118827	13604523	10867793		
8192	1963547	3399327	5234728	7289171	9823226	10315743	10229748	10978233	11160086	11348065	8401313	8191020	
16384	2005892	3444791	5221128	7380615	9067322	9797744	9875182	10246230	10694297	10827411	9448826	8039428	7688629
32768	0	0	0	0	9115315	9642940	9781570	10204999	9230406	10628800	9351620	7928496	7514100
65536	0	0	0	0	9109818	9092040	9640507	10182622	10465636	10458866	9343612	7903603	7514882
131072	0	0	0	0	8841815	9398530	9504143	9893067	10154186	10189002	8780245	7875502	7587778
262144	0	0	0	0	8874486	9362262	9523472	9969385	10249963	10291506	9235650	7783833	7548956
524288	0	0	0	0	8271332	9507447	9722155	10087734	10334688	8518762	9281915	7745063	7524335
Random Read Report													
	4	8	16	32	64	128	256	512	1024	2048	4096	8192	16384
64	2006158	3791156	6421025	10821524	15972885								
128	2175277	3759450	6406138	9795896	14200794	15881078							
256	2061693	3879045	6107548	9868433	13454450	16072608	16072608						
512	2091421	3628697	6018636	9468384	13190469	15583417	15471149	17069844					
1024	2052253	3778101	6059442	9464330	13102174	15516181	15516181	16531459	17056672				
2048	2033695	3812993	6041567	9483197	13219235	15399510	15075199	16525279	17050094	17084004			
4096	2051130	3723578	6060333	9524234	13166611	15341830	12040791	14427096	14889742	14177082	10923071		
8192	1791734	3239404	5171695	6350207	10076743	11620560	11038189	11393220	11788003	10908526	8289830	8234205	
16384	1806989	3183254	4969299	7138346	8771050	9631583	9722887	10292268	10631426	10557924	9464442	8039428	7698966
32768	0	0	0	0	8763671	9492423	9677568	10230066	8825007	10533491	9349076	7934447	7548353
65536	0	0	0	0	8692944	8946740	9571355	10118525	10392828	10407781	9344882	7915894	7645307
131072	0	0	0	0	8500177	9316978	9511049	9892176	10153623	10131542	8825208	7836324	7555035
262144	0	0	0	0	8313585	9091846	9472848	9751955	9971193	10038469	9011444	7631532	7412114
524288	0	0	0	0	7632110	9235798	9487143	9852086	10052456	8413359	9045688	7599142	7398916
Random Write Report													
	4	8	16	32	64	128	256	512	1024	2048	4096	8192	16384
64	809014	1562436	2801873	5389653	6421025								
128	895074	1521718	2969325	5603747	6406138	7917784							
256	828628	1579483	2880164	5022044	6249745	7314033	7965107						
512	837942	1488408	2681653	5067142	6156675	8265729	7871843	7900804					
1024	840041	1585557	2861774	4920874	6364746	8822754	8457895	8750850	8840915				
2048	830522	1553950	2771120	4994712	6829301	9062970	8940345	8866519	9308497	9525260			
4096	829152	1576596	2860067	4922872	7802627	10950922	8906998	9545402	9327731	9202815	7969117		
8192	791866	1480333	2679789	4536055	6959913	10026754	9217148	9071145	9083135	8952964	6810914	6655245	
16384	808133	1539249	2732514	4650496	6826343	9225569	9012620	8983166	8983166	8864963	8282642	6622411	5813926
32768	0	0	0	0	6916279	9435078	9168216	9119549	8044984	9079185	8308582	6750504	5874512
65536	0	0	0	0	6935790	8622050	9152893	8998581	9049238	8960447	8695144	6841706	6053005
131072	0	0	0	0	6849788	9636925	9206973	9211601	9043850	8994135	8342677	6896706	6025196
262144	0	0	0	0	7032922	9554755	9345312	8837679	9242793	9497969	9288941	6880599	6024232
524288	0	0	0	0	6301995	9563972	9137438	9157376	9338598	8522658	9381746	6773753	6114787
Backward Read Report													
	4	8	16	32	64	128	256	512	1024	2048	4096	8192	16384
64	1780008	2561267	4643754	6421025	8182586								
128	1967960	3277486	5122535	7582312	9977956	10779307							
256	1985448	3465855	5569035	8271916	10651598	12812277	13454450						
512	2047551	3533174	5807059	8140399	10911694	13438092	12215097	14628067					
1024	1896395	3735385	5720477	8822754	12208350	13998981	13472053	15516181	15977962				
2048	1872340	3199462	4707314	6340344	11502234	12882215	12729493	13726200	14441580	12345306			
4096	1843372	3083911	4707064	6847827	9242423	9756822	9614850	11704457	11641010	10369272	8981502		
8192	1761694	3007204	4571664	6253130	8659632	9547539	9603578	9965682	10175219	10038471	8159896	7922821	
16384	1819236	3152586	4854508	6835170	8554853	9198402	9491895	9959624	10144884	10138897	8785628	7838599	7610294
32768	0	0	0	0	8726390	9343991	9555781	10039510	10113386	10356489	8782711	7775981	7533045
65536	0	0	0	0	8825235	9244625	9547751	9974955	10314056	10280111	8588108	7798219	7618608
131072	0	0	0	0	8727002	9228456	9469111	3528264	10274502	10177119	8561200	7602363	7377212
262144	0	0	0	0	8578554	8030919	9344598	9888423	10119775	10217673	5178344	7636197	7504389
524288	0	0	0	0	8742942	9299186	9519341	9823654	10090789	8746733	9001256	7720562	5658561
Record Rewrite Report													
	4	8	16	32	64	128	256	512	1024	2048	4096	8192	16384
64	852702	1392258	3057153	5389653	5860307								
128	908709	1598754	3053774	5325799	7082197	8036304							
256	833776	1591186	2842047	5117791	7120034	9434868	8024634						
512	837942	1541840	2845081	4732165	6909408	10235583	8528336	8140399					
1024	815943	1589665	2852271	4926518	7472033	11249091	9677584	9485232	8680108				
2048	814922	1563282	2781889	4900677	7670915	11187641	10351742	10351742	9932791	9483197			
4096	820989	1574718	2830848	5007531	8062616	11348794	10142762	10586522	9897345	9353122	7788478		
8192	798046	1513657	2736997	4934754	7915520	11378128	10734715	10450659	10877445	10967720	6793408	6090202	
16384	815857	1577046	2857262	5063026	8030973	11628043	11018377	11069849	11161547	11252933	8560181	6641612	6005999
32768	0	0	0	0	8350481	12113635	11314822	11295294	10962836	11389837	8708144	6889928	5979556
65536	0	0	0	0	8424651	11870286	11339938	11247139	11471023	11625789	9158077	6831673	6236763
131072	0	0	0	0	8326756	11918875	11478401	11256767	11664468	11658037	8786841	6876778	6152159
262144	0	0	0	0	8571532	11809777	11435877	11153178	11873544	11740546	8958506	6946061	6147715
524288	0	0	0	0	8242623	11960976	11141526	11199404	11307845	11340326	9186757	6979811	6039002
Stride Read Report													
	4	8	16	32	64	128	256	512	1024	2048	4096	8192	16384
64	2067979	3541098	6421025	9318832	12902017								
128	2035099	4012317	5545860	9129573	12842051	11720614							
256	2019049	3756894	6073004	9868433	13454450	15835569	15164624						
512	2065273	3580298	5951911	9859630	12427157	14240069	13872122	13438092					
1024	2008114	3696803	5536137	9142007	13863422	16468073	16280798	16531459	17124680				
2048	1823840	3074635	4521480	6317030	12729493	16653431	16122077	16367837	16783585	12257227			
4096	1799541	3009363	4686519	6872481	12756011	14889742	15880734	16058868	16256412	14318876	9207748		
8192	1738870	2976981	4657175	6258825	9977258	11131163	11009893	16386705	16481026	15367944	7275280	8056577	
16384	1794110	3100107	4793220	6849477	8987865	9514235	9729770	10382458	17048570	16335310	10543345	8098167	7628034
32768	0	0	0	0	8835219	9530601	9749652	10214859	10303692	16915968	11354081	7622452	7493207
65536	0	0	0	0	8767250	9563031	9623294	10056335	10417248	10297827	11638587	8508885	7447286
131072	0	0	0	0	8143106	8406848	9506279	9491180	10209627	10250364	8961587	8608927	7612574
262144	0	0	0	0	8139357	8651590	9286038	9943689	10246811	10171263	8576279	7755065	7638265
524288	0	0	0	0	8212671	9100945	9426749	9707563	10147598	9991209	9106975	7757576	7250448
Fwrite Report													
	4	8	16	32	64	128	256	512	1024	2048	4096	8192	16384
64	863676	1562436	3022727	5389653	6421025								
128	914904	1689305	3053774	5325799	6406138	7582312							
256	850282	1619994	2872459	5117791	6249745	8271916	7965107						
512	861818	1575781	2942541	4571003	6018636	7620440	7871843	7620440					
1024	840041	1628234	2659742	5018624	6692005	8895850	8680108	8970167	8680108				
2048	853377	1594920	2868277	4897882	6945257	9653719	9015410	9228493	9268323	9567698			
4096	859775	1613162	2908979	5020702	7954358	10950922	8569343	9545402	9163546	9327731	7683972		
8192	824156	1529491	2752785	4703073	7236972	10199382	9339916	9526362	8904241	9032989	856812	6046263	
16384	852178	1584062	2783539	4791883	7025234	9460533	9001994	8914412	9164056	8972610	7926299	6499029	5968960
32768	0	0	0	0	7095525	9646324	9132274	9061825	8677903	9004828	7917079	6706692	5910385
65536	0	0	0	0	7211273	9593737	9390534	9108611	9126454	9046855	8290490	6745171	5855591
131072	0	0	0	0	7027177	9324247	9146011	8373173	9133247	9180681	8640181	6702393	6079970
262144	0	0	0	0	7152234	8595454	9198716	9162688	9520751	9094629	8411466	6812474	5972464
524288	0	0	0	0	7032602	9630232	9191557	9211463	9286344	9306939	9253714	6936767	5742537
Re-fwrite Report													
	4	8	16	32	64	128	256	512	1024	2048	4096	8192	16384
64	901377	1526887	2892445	4988978	5860307								
128	907174	1663139	2985839	5122535	5847904	7582312							
256	850282	1600674	2588541	5022044	5810112	7791708	7314033						
512	808904	1614885	2894940	2796910	5886650	7647578	7758090	8109658					
1024	861957	1625154	2884840	4972145	6609617	9142007	8525047	8542002	8680108				
2048	824464	1568134	2694623	4394265	6732948	8866519	8830061	8678404	8820994	8067138			
4096	842612	1574141	2752828	4681411	7024216	9801354	8459632	8638284	7715028	8677553	6978564		
8192	823149	1514525	2708945	4551075	6959913	8971666	9194949	8376734	9234488	8772387	6385612	6114043	
16384	833409	1581000	2805471	4771256	6939394	9443632	9086505	9001994	9042267	8809279	7606924	6370107	5887143
32768	0	0	0	0	7039196	9539200	9238472	9254023	8731379	8913722	7724845	6537923	5933093
65536	0	0	0	0	7182819	9186541	9237169	9092040	9003296	9066849	8111889	6643927	5890980
131072	0	0	0	0	6192282	9420917	9201272	8833574	9058901	9091560	8265413	6422920	6022951
262144	0	0	0	0	7085769	9208963	9031059	8913118	4416014	8966177	8638200	6790596	6000102
524288	0	0	0	0	7041407	9714596	9284031	9182423	9178245	9037881	9181464	3334228	5942570
Fread Report													
	4	8	16	32	64	128	256	512	1024	2048	4096	8192	16384
64	1828508	2772930	3203069	3363612	5389653								
128	2098744	3445772	4934216	6114306	5784891	3359523							
256	1911233	3499745	4496299	5841722	7314033	6891544	3236056						
512	2105777	3325278	4650188	6174377	7871843	7758090	6551087	3220553					
1024	2077065	3567824	4570058	6559147	8262639	8391792	7524394	6963241	3390391				
2048	1877250	3093459	4112366	5006356	7999524	8190205	7609754	6990474	6243566	3007884			
4096	1855917	2970339	4157956	5411313	6532752	6804432	6597984	6449370	5353976	5153230	1975877		
8192	1767857	2916338	4122934	5062726	6364322	6655245	6229323	6186701	6281710	5610847	2719666	1920310	
16384	1851887	3032391	4310375	5457909	6402750	6496572	6298872	6407526	6387277	5942635	3338845	2611437	1914468
32768	0	0	0	0	6474477	6566975	6414945	6517152	6557262	6111166	4147371	3168781	2561206
65536	0	0	0	0	6519131	6587245	6425112	6534629	6539137	2081370	4776680	3560972	3049157
131072	0	0	0	0	6475882	6585573	6398178	6505846	6522906	6136845	5046232	3735520	2214053
262144	0	0	0	0	6435055	6579418	6357145	6529135	6625806	6262568	5151703	3930718	3619424
524288	0	0	0	0	6503122	6502622	6444764	6480527	6531807	5062163	4365031	2886142	3745793
Re-fread Report													
	4	8	16	32	64	128	256	512	1024	2048	4096	8192	16384
64	2203800	3791156	5860307	7100397	9006179								
128	2098744	3657016	5603747	7582312	8548124	8548124							
256	2045979	3717869	5320671	7073132	8534922	8815202	8534922						
512	2124526	3634839	5177083	6843354	8665997	8808176	8528336	8844453					
1024	2102484	3668386	4898425	7008693	8750850	9064828	8542002	8750850	8895850				
2048	2083516	3617110	5107566	6917293	8634786	8678404	8574451	8758039	8857376	7135648			
4096	2012682	3465216	4946971	6680080	8193352	8788530	7486415	7877762	7355007	6351609	4680135		
8192	1817994	3047749	4425630	5494212	6906749	7111158	6638529	6643663	6709830	6215800	4983423	3972772	
16384	1866930	3082585	4384071	5597504	6504566	6687502	6445388	6561079	6609036	6277582	5178244	3941284	3996990
32768	0	0	0	0	6567916	6637380	6479666	6627778	6658926	6297950	5314360	4071564	3772084
65536	0	0	0	0	6541782	6616579	6466984	6581883	6615783	6020788	5352942	4085775	3898822
131072	0	0	0	0	6575019	6624138	6420670	6535002	6569990	6400338	5403068	4054820	3782867
262144	0	0	0	0	6494848	6627843	6408127	6573989	6652063	6507726	5361488	4054503	3883844
524288	0	0	0	0	6510323	6536933	6457463	6503526	6575951	6156800	5228402	2944563	3889750
 
madrebel said:
how are those 6 drives configured?
Click to expand...

1 x RaidZ2 vdev, as mentioned in the initial post.

Code: 
[root@freenas] ~# zpool status
  pool: store
 state: ONLINE
  scan: none requested
config:

        NAME                                            STATE     READ WRITE CKSUM
        store                                           ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/1dbdf422-e60c-11e2-951b-bc5ff4b12137  ONLINE       0     0     0
            gptid/1e0aece6-e60c-11e2-951b-bc5ff4b12137  ONLINE       0     0     0
            gptid/1e591b48-e60c-11e2-951b-bc5ff4b12137  ONLINE       0     0     0
            gptid/1ea5d49e-e60c-11e2-951b-bc5ff4b12137  ONLINE       0     0     0
            gptid/1ef543e5-e60c-11e2-951b-bc5ff4b12137  ONLINE       0     0     0
            gptid/1f426666-e60c-11e2-951b-bc5ff4b12137  ONLINE       0     0     0

errors: No known data errors
 
sorry missed that.

given that though i don't see any egregious outliers. you can tweak a little here and there to improve things but by and large your performance is fairly good.

BSD smb isn't the best performing from what i've read. if you tweak the network buffers there (smb.conf) you main gain a bit better sustained throughput.

have you done anything in ... what is it in the bsd world again /boot/loader.conf? you may want to play around with some of the ZFS tuneables. i wouldn't expect a great deal more than youre getting though for a single 6 disk z2 vdev. crystaldisk as an example is already maxing out the single gig link for sequential throughput. I imagine you'll get closer to line speed after adjusting smb.conf a bit.
 
Yes. Although I can't remember if I ran tests before or after altering loader.conf and sysctl.conf. I let autotune change a few things via the FreeNAS GUI.

sysctl.conf
Code: 
kern.ipc.maxsockbuf=2097152
net.inet.tcp.recvbuf_max=2097152
net.inet.tcp.sendbuf_max=2097152

loader.conf
Code: 
vfs.zfs.arc_max=10362036970
vm.kmem_size=11513374412
vm.kmem_size_max=14391718016

I'm a bit confused by something related to this though. If I look in either of the files, I don't see these settings in there. Does FreeNAS store these values somewhere? How are they loaded?


I don't think I understand the iozone results. 7,000,000 - 10,000,000 Kbytes/sec would be 7-10 Gigabytes/sec correct? Is that...wrong?


PS: Yes, I've also read and heard that a Solaris based system should outperform a FreeBSD based system. I will probably move to that, but for simplicity FreeNAS is easier for me to get up and running to perform some testing on. But considering everything should be v28 I should be able to move between FreeBSD, Solaris derivatives, and ZoL.
 
It is Gigabytes per second,

-l doesn't work with all systems, the super high results are reporting either CPU cache or RAM cache
 
staticlag said:
It is Gigabytes per second,

-l doesn't work with all systems, the super high results are reporting either CPU cache or RAM cache
Click to expand...

Blah. See, I was trying to avoid the buffer cache with the -I. What would be the recommended way to avoid cache if -I doesn't work?
 
What about -U ?

-U mountpoint
Mount point to unmount and remount between tests. Iozone will unmount and remount
this mount point before beginning each test. This guarantees that the buffer cache does not
contain any of the file under test.
Click to expand...

iozone.org/docs
 
zfs is pretty advanced, its pretty hard to fool it's cache with something other than simply a huge file.
 
Yea, the more I'm reading the more I'm coming to that same consensus. I found THIS post over on the FreeNAS forums helpful.

I guess, if nothing appears "wrong" then I should assume things are good.
I wanted to use something more advanced then a dd to see actual disk performance minus the caching.
I'm not sure if I should even bother thrashing the system with a large iozone test.

Even if I use a large file, would it still be possible that I'd still hit some of the cache? Would I have to monitor it (with arcstat)?
 
I'm not gonna lie, your numbers look good for what you have.

A single raidz2 configuration is nearly the slowest and the worst performing configuration you can have.

By adding another vdev of an identical raidz2 pool your benchmarks will be through the roof.
 
not likely. his random IO would in theory double however he is already seeing about 200x the random IO he would be seeing if there was no caching going on.

then for sequential he is already maxing out the gig link so there isn't anything to be gained their either.
 
You must log in or register to reply here.
Back
Top