ZFS performance test with ZFSguru

B

bao__zhe

Limp Gawd
Joined
Apr 18, 2012
Messages
135
So I finally assembled my rig for a ZFS storage server. And here are the performance figures:

benchseqwrite.png


benchseqread.png



Code: 
ZFSGURU-benchmark, version 1
Test size: 32.000 gigabytes (GiB)
Test rounds: 5
Cooldown period: 2 seconds
Sector size override: 4096 bytes
Number of disks: 10 disks
disk 1: label/10.nop
disk 2: label/11.nop
disk 3: label/12.nop
disk 4: label/13.nop
disk 5: label/14.nop
disk 6: label/3.nop
disk 7: label/6.nop
disk 8: label/7.nop
disk 9: label/8.nop
disk 10: label/9.nop


* Test Settings: TS32; TR5; SECT4096; 
* Tuning: none
* Stopping background processes: sendmail, moused, syslogd and cron
* Stopping Samba service

Now testing RAID0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	651 MiB/sec	638 MiB/sec	641 MiB/sec	652 MiB/sec	652 MiB/sec	= 647 MiB/sec avg
WRITE:	892 MiB/sec	576 MiB/sec	890 MiB/sec	896 MiB/sec	890 MiB/sec	= 829 MiB/sec avg

Now testing RAID0 configuration with 9 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	917 MiB/sec	913 MiB/sec	904 MiB/sec	913 MiB/sec	913 MiB/sec	= 912 MiB/sec avg
WRITE:	955 MiB/sec	979 MiB/sec	978 MiB/sec	977 MiB/sec	961 MiB/sec	= 970 MiB/sec avg

Now testing RAID0 configuration with 10 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	1198 MiB/sec	1193 MiB/sec	1189 MiB/sec	1213 MiB/sec	1188 MiB/sec	= 1196 MiB/sec avg
WRITE:	1040 MiB/sec	1042 MiB/sec	1032 MiB/sec	1041 MiB/sec	1044 MiB/sec	= 1040 MiB/sec avg

Now testing RAIDZ configuration with 8 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	720 MiB/sec	722 MiB/sec	715 MiB/sec	723 MiB/sec	723 MiB/sec	= 721 MiB/sec avg
WRITE:	637 MiB/sec	644 MiB/sec	646 MiB/sec	637 MiB/sec	624 MiB/sec	= 638 MiB/sec avg

Now testing RAIDZ configuration with 9 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	735 MiB/sec	749 MiB/sec	743 MiB/sec	745 MiB/sec	734 MiB/sec	= 741 MiB/sec avg
WRITE:	721 MiB/sec	718 MiB/sec	724 MiB/sec	719 MiB/sec	715 MiB/sec	= 720 MiB/sec avg

Now testing RAIDZ configuration with 10 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	634 MiB/sec	630 MiB/sec	632 MiB/sec	636 MiB/sec	644 MiB/sec	= 635 MiB/sec avg
WRITE:	740 MiB/sec	763 MiB/sec	767 MiB/sec	767 MiB/sec	767 MiB/sec	= 761 MiB/sec avg

Now testing RAIDZ2 configuration with 8 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	677 MiB/sec	677 MiB/sec	680 MiB/sec	672 MiB/sec	673 MiB/sec	= 676 MiB/sec avg
WRITE:	487 MiB/sec	463 MiB/sec	520 MiB/sec	467 MiB/sec	519 MiB/sec	= 491 MiB/sec avg

Now testing RAIDZ2 configuration with 9 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	680 MiB/sec	669 MiB/sec	677 MiB/sec	672 MiB/sec	676 MiB/sec	= 675 MiB/sec avg
WRITE:	527 MiB/sec	518 MiB/sec	506 MiB/sec	503 MiB/sec	521 MiB/sec	= 515 MiB/sec avg

Now testing RAIDZ2 configuration with 10 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	711 MiB/sec	719 MiB/sec	704 MiB/sec	702 MiB/sec	709 MiB/sec	= 709 MiB/sec avg
WRITE:	527 MiB/sec	575 MiB/sec	527 MiB/sec	558 MiB/sec	538 MiB/sec	= 545 MiB/sec avg

Now testing RAID1 configuration with 8 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	612 MiB/sec	612 MiB/sec	613 MiB/sec	607 MiB/sec	620 MiB/sec	= 613 MiB/sec avg
WRITE:	114 MiB/sec	89 MiB/sec	107 MiB/sec	104 MiB/sec	114 MiB/sec	= 105 MiB/sec avg

Now testing RAID1 configuration with 9 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	579 MiB/sec	577 MiB/sec	603 MiB/sec	595 MiB/sec	583 MiB/sec	= 587 MiB/sec avg
WRITE:	113 MiB/sec	114 MiB/sec	98 MiB/sec	95 MiB/sec	91 MiB/sec	= 102 MiB/sec avg

Now testing RAID1 configuration with 10 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	608 MiB/sec	586 MiB/sec	583 MiB/sec	595 MiB/sec	586 MiB/sec	= 592 MiB/sec avg
WRITE:	99 MiB/sec	89 MiB/sec	112 MiB/sec	89 MiB/sec	91 MiB/sec	= 96 MiB/sec avg

Now testing RAID1+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	607 MiB/sec	596 MiB/sec	595 MiB/sec	600 MiB/sec	611 MiB/sec	= 602 MiB/sec avg
WRITE:	456 MiB/sec	541 MiB/sec	458 MiB/sec	574 MiB/sec	485 MiB/sec	= 503 MiB/sec avg

Now testing RAID1+0 configuration with 10 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	644 MiB/sec	648 MiB/sec	640 MiB/sec	646 MiB/sec	650 MiB/sec	= 646 MiB/sec avg
WRITE:	516 MiB/sec	477 MiB/sec	472 MiB/sec	600 MiB/sec	604 MiB/sec	= 534 MiB/sec avg

Now testing RAIDZ+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	682 MiB/sec	673 MiB/sec	686 MiB/sec	683 MiB/sec	674 MiB/sec	= 679 MiB/sec avg
WRITE:	587 MiB/sec	591 MiB/sec	587 MiB/sec	573 MiB/sec	586 MiB/sec	= 585 MiB/sec avg

Now testing RAID0 configuration with 4 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	485 MiB/sec	488 MiB/sec	482 MiB/sec	483 MiB/sec	484 MiB/sec	= 484 MiB/sec avg
WRITE:	498 MiB/sec	449 MiB/sec	433 MiB/sec	528 MiB/sec	447 MiB/sec	= 471 MiB/sec avg

Now testing RAID0 configuration with 5 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	531 MiB/sec	533 MiB/sec	543 MiB/sec	527 MiB/sec	527 MiB/sec	= 532 MiB/sec avg
WRITE:	487 MiB/sec	498 MiB/sec	489 MiB/sec	631 MiB/sec	622 MiB/sec	= 545 MiB/sec avg

Now testing RAID0 configuration with 6 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	630 MiB/sec	631 MiB/sec	634 MiB/sec	634 MiB/sec	630 MiB/sec	= 632 MiB/sec avg
WRITE:	711 MiB/sec	706 MiB/sec	713 MiB/sec	711 MiB/sec	708 MiB/sec	= 710 MiB/sec avg

Now testing RAID0 configuration with 7 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	737 MiB/sec	705 MiB/sec	725 MiB/sec	703 MiB/sec	720 MiB/sec	= 718 MiB/sec avg
WRITE:	808 MiB/sec	795 MiB/sec	817 MiB/sec	813 MiB/sec	698 MiB/sec	= 786 MiB/sec avg

Now testing RAIDZ configuration with 4 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	418 MiB/sec	402 MiB/sec	419 MiB/sec	402 MiB/sec	398 MiB/sec	= 408 MiB/sec avg
WRITE:	357 MiB/sec	300 MiB/sec	315 MiB/sec	298 MiB/sec	328 MiB/sec	= 320 MiB/sec avg

Now testing RAIDZ configuration with 5 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	543 MiB/sec	538 MiB/sec	537 MiB/sec	541 MiB/sec	539 MiB/sec	= 539 MiB/sec avg
WRITE:	457 MiB/sec	457 MiB/sec	523 MiB/sec	505 MiB/sec	454 MiB/sec	= 479 MiB/sec avg

Now testing RAIDZ configuration with 6 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	578 MiB/sec	579 MiB/sec	581 MiB/sec	578 MiB/sec	577 MiB/sec	= 579 MiB/sec avg
WRITE:	507 MiB/sec	510 MiB/sec	520 MiB/sec	499 MiB/sec	510 MiB/sec	= 509 MiB/sec avg

Now testing RAIDZ configuration with 7 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	591 MiB/sec	587 MiB/sec	598 MiB/sec	594 MiB/sec	591 MiB/sec	= 592 MiB/sec avg
WRITE:	586 MiB/sec	586 MiB/sec	592 MiB/sec	584 MiB/sec	562 MiB/sec	= 582 MiB/sec avg

Now testing RAIDZ2 configuration with 4 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	287 MiB/sec	289 MiB/sec	286 MiB/sec	288 MiB/sec	290 MiB/sec	= 288 MiB/sec avg
WRITE:	208 MiB/sec	204 MiB/sec	205 MiB/sec	205 MiB/sec	224 MiB/sec	= 209 MiB/sec avg

Now testing RAIDZ2 configuration with 5 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	393 MiB/sec	394 MiB/sec	401 MiB/sec	389 MiB/sec	395 MiB/sec	= 394 MiB/sec avg
WRITE:	323 MiB/sec	314 MiB/sec	336 MiB/sec	333 MiB/sec	328 MiB/sec	= 327 MiB/sec avg

Now testing RAIDZ2 configuration with 6 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	490 MiB/sec	500 MiB/sec	493 MiB/sec	486 MiB/sec	488 MiB/sec	= 492 MiB/sec avg
WRITE:	391 MiB/sec	386 MiB/sec	385 MiB/sec	382 MiB/sec	382 MiB/sec	= 385 MiB/sec avg

Now testing RAIDZ2 configuration with 7 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	601 MiB/sec	599 MiB/sec	602 MiB/sec	600 MiB/sec	601 MiB/sec	= 601 MiB/sec avg
WRITE:	413 MiB/sec	414 MiB/sec	416 MiB/sec	412 MiB/sec	417 MiB/sec	= 414 MiB/sec avg

Now testing RAID1 configuration with 4 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	409 MiB/sec	416 MiB/sec	412 MiB/sec	414 MiB/sec	416 MiB/sec	= 413 MiB/sec avg
WRITE:	111 MiB/sec	99 MiB/sec	80 MiB/sec	116 MiB/sec	92 MiB/sec	= 99 MiB/sec avg

Now testing RAID1 configuration with 5 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	497 MiB/sec	489 MiB/sec	489 MiB/sec	485 MiB/sec	486 MiB/sec	= 489 MiB/sec avg
WRITE:	108 MiB/sec	111 MiB/sec	88 MiB/sec	89 MiB/sec	92 MiB/sec	= 98 MiB/sec avg

Now testing RAID1 configuration with 6 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	591 MiB/sec	588 MiB/sec	594 MiB/sec	598 MiB/sec	590 MiB/sec	= 592 MiB/sec avg
WRITE:	84 MiB/sec	94 MiB/sec	106 MiB/sec	109 MiB/sec	86 MiB/sec	= 96 MiB/sec avg

Now testing RAID1 configuration with 7 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	681 MiB/sec	678 MiB/sec	682 MiB/sec	679 MiB/sec	691 MiB/sec	= 682 MiB/sec avg
WRITE:	106 MiB/sec	101 MiB/sec	114 MiB/sec	94 MiB/sec	111 MiB/sec	= 105 MiB/sec avg

Now testing RAID1+0 configuration with 4 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	430 MiB/sec	423 MiB/sec	422 MiB/sec	411 MiB/sec	432 MiB/sec	= 423 MiB/sec avg
WRITE:	276 MiB/sec	226 MiB/sec	242 MiB/sec	234 MiB/sec	257 MiB/sec	= 247 MiB/sec avg

Now testing RAID1+0 configuration with 6 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	540 MiB/sec	541 MiB/sec	539 MiB/sec	541 MiB/sec	537 MiB/sec	= 540 MiB/sec avg
WRITE:	396 MiB/sec	398 MiB/sec	353 MiB/sec	356 MiB/sec	364 MiB/sec	= 373 MiB/sec avg

Now testing RAIDZ+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	676 MiB/sec	675 MiB/sec	677 MiB/sec	669 MiB/sec	675 MiB/sec	= 408 MiB/sec avg
WRITE:	580 MiB/sec	578 MiB/sec	585 MiB/sec	588 MiB/sec	576 MiB/sec	= 320 MiB/sec avg

Now testing RAID0 configuration with 1 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	183 MiB/sec	182 MiB/sec	177 MiB/sec	176 MiB/sec	187 MiB/sec	= 181 MiB/sec avg
WRITE:	98 MiB/sec	112 MiB/sec	99 MiB/sec	107 MiB/sec	82 MiB/sec	= 100 MiB/sec avg

Now testing RAID0 configuration with 2 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	286 MiB/sec	303 MiB/sec	307 MiB/sec	283 MiB/sec	317 MiB/sec	= 299 MiB/sec avg
WRITE:	237 MiB/sec	281 MiB/sec	250 MiB/sec	253 MiB/sec	269 MiB/sec	= 258 MiB/sec avg

Now testing RAID0 configuration with 3 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	405 MiB/sec	389 MiB/sec	407 MiB/sec	397 MiB/sec	392 MiB/sec	= 398 MiB/sec avg
WRITE:	420 MiB/sec	333 MiB/sec	419 MiB/sec	364 MiB/sec	331 MiB/sec	= 373 MiB/sec avg

Now testing RAIDZ configuration with 2 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	180 MiB/sec	180 MiB/sec	177 MiB/sec	180 MiB/sec	177 MiB/sec	= 179 MiB/sec avg
WRITE:	116 MiB/sec	104 MiB/sec	122 MiB/sec	114 MiB/sec	129 MiB/sec	= 117 MiB/sec avg

Now testing RAIDZ configuration with 3 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	323 MiB/sec	320 MiB/sec	322 MiB/sec	332 MiB/sec	335 MiB/sec	= 326 MiB/sec avg
WRITE:	254 MiB/sec	194 MiB/sec	269 MiB/sec	257 MiB/sec	201 MiB/sec	= 235 MiB/sec avg

Now testing RAIDZ2 configuration with 3 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	164 MiB/sec	169 MiB/sec	165 MiB/sec	169 MiB/sec	169 MiB/sec	= 167 MiB/sec avg
WRITE:	119 MiB/sec	134 MiB/sec	118 MiB/sec	121 MiB/sec	137 MiB/sec	= 126 MiB/sec avg

Now testing RAID1 configuration with 2 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	224 MiB/sec	223 MiB/sec	225 MiB/sec	225 MiB/sec	224 MiB/sec	= 224 MiB/sec avg
WRITE:	103 MiB/sec	107 MiB/sec	104 MiB/sec	81 MiB/sec	104 MiB/sec	= 100 MiB/sec avg

Now testing RAID1 configuration with 3 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	337 MiB/sec	335 MiB/sec	336 MiB/sec	333 MiB/sec	334 MiB/sec	= 335 MiB/sec avg
WRITE:	81 MiB/sec	78 MiB/sec	85 MiB/sec	110 MiB/sec	110 MiB/sec	= 93 MiB/sec avg

Now testing RAIDZ+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@cWmRd@cWmRd@
READ:	669 MiB/sec	672 MiB/sec	678 MiB/sec	678 MiB/sec	670 MiB/sec	= 408 MiB/sec avg
WRITE:	581 MiB/sec	585 MiB/sec	592 MiB/sec	580 MiB/sec	583 MiB/sec	= 320 MiB/sec avg

Done

so what do you think? looks normal? any suggestions?

my config:

ESXi on Xeon E5-1650 & 48GB REG ECC DDR3-1600 RAM
ZFSguru VM with 1 core, 16GB RAM, LSI 9201-16i PCI passthrough
10x Seagate ST3000DM001-1CH166

PS: working on benchmark with no 4K override now.
 
Last edited:
Thats pretty fast from what I can tell. I am building almost the same server... with 8 wd reds Xeon E5-1620 and 32GB ECC 1600 Registered.

Except I am not using a VM, I am going to go full on physical.

I have all my parts in except the processor which will arrive tomorrow.
 
And here is the performance with 512B sector size by default.

benchseqwrite.png


benchseqread.png


Code: 
ZFSGURU-benchmark, version 1
Test size: 32.000 gigabytes (GiB)
Test rounds: 3
Cooldown period: 2 seconds
Sector size override: default (no override)
Number of disks: 10 disks
disk 1: label/10
disk 2: label/11
disk 3: label/12
disk 4: label/13
disk 5: label/14
disk 6: label/3
disk 7: label/6
disk 8: label/7
disk 9: label/8
disk 10: label/9


* Test Settings: TS32; 
* Tuning: none
* Stopping background processes: sendmail, moused, syslogd and cron
* Stopping Samba service

Now testing RAID0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
READ:	630 MiB/sec	632 MiB/sec	630 MiB/sec	= 631 MiB/sec avg
WRITE:	775 MiB/sec	776 MiB/sec	799 MiB/sec	= 783 MiB/sec avg

Now testing RAID0 configuration with 9 disks: cWmRd@cWmRd@cWmRd@
READ:	852 MiB/sec	911 MiB/sec	924 MiB/sec	= 896 MiB/sec avg
WRITE:	869 MiB/sec	843 MiB/sec	825 MiB/sec	= 845 MiB/sec avg

Now testing RAID0 configuration with 10 disks: cWmRd@cWmRd@cWmRd@
READ:	1193 MiB/sec	1192 MiB/sec	1197 MiB/sec	= 1194 MiB/sec avg
WRITE:	852 MiB/sec	908 MiB/sec	902 MiB/sec	= 887 MiB/sec avg

Now testing RAIDZ configuration with 8 disks: cWmRd@cWmRd@cWmRd@
READ:	608 MiB/sec	598 MiB/sec	608 MiB/sec	= 605 MiB/sec avg
WRITE:	563 MiB/sec	566 MiB/sec	582 MiB/sec	= 570 MiB/sec avg

Now testing RAIDZ configuration with 9 disks: cWmRd@cWmRd@cWmRd@
READ:	628 MiB/sec	610 MiB/sec	613 MiB/sec	= 617 MiB/sec avg
WRITE:	604 MiB/sec	630 MiB/sec	611 MiB/sec	= 615 MiB/sec avg

Now testing RAIDZ configuration with 10 disks: cWmRd@cWmRd@cWmRd@
READ:	660 MiB/sec	660 MiB/sec	614 MiB/sec	= 645 MiB/sec avg
WRITE:	612 MiB/sec	643 MiB/sec	638 MiB/sec	= 631 MiB/sec avg

Now testing RAIDZ2 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
READ:	618 MiB/sec	628 MiB/sec	645 MiB/sec	= 630 MiB/sec avg
WRITE:	423 MiB/sec	478 MiB/sec	433 MiB/sec	= 445 MiB/sec avg

Now testing RAIDZ2 configuration with 9 disks: cWmRd@cWmRd@cWmRd@
READ:	477 MiB/sec	484 MiB/sec	480 MiB/sec	= 480 MiB/sec avg
WRITE:	468 MiB/sec	464 MiB/sec	471 MiB/sec	= 468 MiB/sec avg

Now testing RAIDZ2 configuration with 10 disks: cWmRd@cWmRd@cWmRd@
READ:	613 MiB/sec	597 MiB/sec	605 MiB/sec	= 605 MiB/sec avg
WRITE:	474 MiB/sec	479 MiB/sec	503 MiB/sec	= 485 MiB/sec avg

Now testing RAID1 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
READ:	582 MiB/sec	574 MiB/sec	598 MiB/sec	= 585 MiB/sec avg
WRITE:	93 MiB/sec	120 MiB/sec	82 MiB/sec	= 98 MiB/sec avg

Now testing RAID1 configuration with 9 disks: cWmRd@cWmRd@cWmRd@
READ:	566 MiB/sec	562 MiB/sec	579 MiB/sec	= 569 MiB/sec avg
WRITE:	89 MiB/sec	101 MiB/sec	101 MiB/sec	= 97 MiB/sec avg

Now testing RAID1 configuration with 10 disks: cWmRd@cWmRd@cWmRd@
READ:	585 MiB/sec	604 MiB/sec	596 MiB/sec	= 595 MiB/sec avg
WRITE:	90 MiB/sec	101 MiB/sec	101 MiB/sec	= 97 MiB/sec avg

Now testing RAID1+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
READ:	611 MiB/sec	612 MiB/sec	603 MiB/sec	= 609 MiB/sec avg
WRITE:	411 MiB/sec	386 MiB/sec	386 MiB/sec	= 395 MiB/sec avg

Now testing RAID1+0 configuration with 10 disks: cWmRd@cWmRd@cWmRd@
READ:	628 MiB/sec	649 MiB/sec	630 MiB/sec	= 636 MiB/sec avg
WRITE:	424 MiB/sec	446 MiB/sec	537 MiB/sec	= 469 MiB/sec avg

Now testing RAIDZ+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
READ:	511 MiB/sec	519 MiB/sec	522 MiB/sec	= 518 MiB/sec avg
WRITE:	545 MiB/sec	532 MiB/sec	545 MiB/sec	= 541 MiB/sec avg

Now testing RAID0 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
READ:	487 MiB/sec	498 MiB/sec	498 MiB/sec	= 494 MiB/sec avg
WRITE:	406 MiB/sec	528 MiB/sec	444 MiB/sec	= 459 MiB/sec avg

Now testing RAID0 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
READ:	522 MiB/sec	514 MiB/sec	521 MiB/sec	= 519 MiB/sec avg
WRITE:	570 MiB/sec	561 MiB/sec	633 MiB/sec	= 588 MiB/sec avg

Now testing RAID0 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
READ:	628 MiB/sec	626 MiB/sec	626 MiB/sec	= 627 MiB/sec avg
WRITE:	498 MiB/sec	484 MiB/sec	493 MiB/sec	= 492 MiB/sec avg

Now testing RAID0 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
READ:	703 MiB/sec	696 MiB/sec	675 MiB/sec	= 691 MiB/sec avg
WRITE:	712 MiB/sec	743 MiB/sec	737 MiB/sec	= 731 MiB/sec avg

Now testing RAIDZ configuration with 4 disks: cWmRd@cWmRd@cWmRd@
READ:	389 MiB/sec	380 MiB/sec	390 MiB/sec	= 387 MiB/sec avg
WRITE:	301 MiB/sec	300 MiB/sec	312 MiB/sec	= 304 MiB/sec avg

Now testing RAIDZ configuration with 5 disks: cWmRd@cWmRd@cWmRd@
READ:	473 MiB/sec	470 MiB/sec	459 MiB/sec	= 467 MiB/sec avg
WRITE:	403 MiB/sec	308 MiB/sec	317 MiB/sec	= 342 MiB/sec avg

Now testing RAIDZ configuration with 6 disks: cWmRd@cWmRd@cWmRd@
READ:	500 MiB/sec	462 MiB/sec	491 MiB/sec	= 484 MiB/sec avg
WRITE:	511 MiB/sec	471 MiB/sec	486 MiB/sec	= 489 MiB/sec avg

Now testing RAIDZ configuration with 7 disks: cWmRd@cWmRd@cWmRd@
READ:	524 MiB/sec	529 MiB/sec	529 MiB/sec	= 527 MiB/sec avg
WRITE:	521 MiB/sec	547 MiB/sec	506 MiB/sec	= 525 MiB/sec avg

Now testing RAIDZ2 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
READ:	280 MiB/sec	268 MiB/sec	271 MiB/sec	= 273 MiB/sec avg
WRITE:	212 MiB/sec	208 MiB/sec	199 MiB/sec	= 206 MiB/sec avg

Now testing RAIDZ2 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
READ:	365 MiB/sec	377 MiB/sec	395 MiB/sec	= 379 MiB/sec avg
WRITE:	290 MiB/sec	233 MiB/sec	315 MiB/sec	= 279 MiB/sec avg

Now testing RAIDZ2 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
READ:	421 MiB/sec	404 MiB/sec	399 MiB/sec	= 408 MiB/sec avg
WRITE:	386 MiB/sec	304 MiB/sec	409 MiB/sec	= 366 MiB/sec avg

Now testing RAIDZ2 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
READ:	541 MiB/sec	512 MiB/sec	518 MiB/sec	= 524 MiB/sec avg
WRITE:	399 MiB/sec	420 MiB/sec	410 MiB/sec	= 409 MiB/sec avg

Now testing RAID1 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
READ:	425 MiB/sec	407 MiB/sec	406 MiB/sec	= 413 MiB/sec avg
WRITE:	97 MiB/sec	96 MiB/sec	88 MiB/sec	= 93 MiB/sec avg

Now testing RAID1 configuration with 5 disks: cWmRd@cWmRd@cWmRd@
READ:	495 MiB/sec	487 MiB/sec	507 MiB/sec	= 497 MiB/sec avg
WRITE:	102 MiB/sec	108 MiB/sec	94 MiB/sec	= 101 MiB/sec avg

Now testing RAID1 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
READ:	587 MiB/sec	587 MiB/sec	581 MiB/sec	= 585 MiB/sec avg
WRITE:	100 MiB/sec	88 MiB/sec	112 MiB/sec	= 100 MiB/sec avg

Now testing RAID1 configuration with 7 disks: cWmRd@cWmRd@cWmRd@
READ:	679 MiB/sec	675 MiB/sec	675 MiB/sec	= 676 MiB/sec avg
WRITE:	102 MiB/sec	120 MiB/sec	97 MiB/sec	= 106 MiB/sec avg

Now testing RAID1+0 configuration with 4 disks: cWmRd@cWmRd@cWmRd@
READ:	413 MiB/sec	407 MiB/sec	436 MiB/sec	= 419 MiB/sec avg
WRITE:	229 MiB/sec	225 MiB/sec	227 MiB/sec	= 227 MiB/sec avg

Now testing RAID1+0 configuration with 6 disks: cWmRd@cWmRd@cWmRd@
READ:	545 MiB/sec	536 MiB/sec	536 MiB/sec	= 539 MiB/sec avg
WRITE:	315 MiB/sec	325 MiB/sec	334 MiB/sec	= 324 MiB/sec avg

Now testing RAIDZ+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
READ:	506 MiB/sec	517 MiB/sec	511 MiB/sec	= 387 MiB/sec avg
WRITE:	466 MiB/sec	540 MiB/sec	539 MiB/sec	= 304 MiB/sec avg

Now testing RAID0 configuration with 1 disks: cWmRd@cWmRd@cWmRd@
READ:	176 MiB/sec	182 MiB/sec	178 MiB/sec	= 179 MiB/sec avg
WRITE:	109 MiB/sec	92 MiB/sec	105 MiB/sec	= 102 MiB/sec avg

Now testing RAID0 configuration with 2 disks: cWmRd@cWmRd@cWmRd@
READ:	298 MiB/sec	300 MiB/sec	300 MiB/sec	= 299 MiB/sec avg
WRITE:	221 MiB/sec	269 MiB/sec	240 MiB/sec	= 243 MiB/sec avg

Now testing RAID0 configuration with 3 disks: cWmRd@cWmRd@cWmRd@
READ:	397 MiB/sec	388 MiB/sec	387 MiB/sec	= 391 MiB/sec avg
WRITE:	294 MiB/sec	302 MiB/sec	351 MiB/sec	= 315 MiB/sec avg

Now testing RAIDZ configuration with 2 disks: cWmRd@cWmRd@cWmRd@
READ:	181 MiB/sec	180 MiB/sec	182 MiB/sec	= 181 MiB/sec avg
WRITE:	95 MiB/sec	117 MiB/sec	92 MiB/sec	= 101 MiB/sec avg

Now testing RAIDZ configuration with 3 disks: cWmRd@cWmRd@cWmRd@
READ:	273 MiB/sec	283 MiB/sec	286 MiB/sec	= 281 MiB/sec avg
WRITE:	222 MiB/sec	208 MiB/sec	230 MiB/sec	= 220 MiB/sec avg

Now testing RAIDZ2 configuration with 3 disks: cWmRd@cWmRd@cWmRd@
READ:	161 MiB/sec	164 MiB/sec	166 MiB/sec	= 164 MiB/sec avg
WRITE:	110 MiB/sec	122 MiB/sec	113 MiB/sec	= 115 MiB/sec avg

Now testing RAID1 configuration with 2 disks: cWmRd@cWmRd@cWmRd@
READ:	225 MiB/sec	225 MiB/sec	227 MiB/sec	= 226 MiB/sec avg
WRITE:	85 MiB/sec	93 MiB/sec	119 MiB/sec	= 99 MiB/sec avg

Now testing RAID1 configuration with 3 disks: cWmRd@cWmRd@cWmRd@
READ:	337 MiB/sec	337 MiB/sec	335 MiB/sec	= 336 MiB/sec avg
WRITE:	76 MiB/sec	104 MiB/sec	101 MiB/sec	= 94 MiB/sec avg

Now testing RAIDZ+0 configuration with 8 disks: cWmRd@cWmRd@cWmRd@
READ:	501 MiB/sec	511 MiB/sec	507 MiB/sec	= 387 MiB/sec avg
WRITE:	518 MiB/sec	589 MiB/sec	532 MiB/sec	= 304 MiB/sec avg

Done

It seems the performance is lower by some amount if not using 4K override.
 
Last edited:
If the disk are 4k sector from the factory I always force 4kb sectors as it does seem to be quicker.
 
Thanks for this, its really going to help with my hardware choices. I'm specifically looking at 4 disk and its interesting that RAIDZ outperforms RAID10.

tangoseal are you going to perform similar tests with the WD reds?
 
Something sounds broken with the test. There is no way raidz should be faster than raid10.
 
It most certainly is faster at sequential read/write. Significantly so in some configurations. I'm doing quite a bit of testing with ZuesRAM and finding that even under VM workloads sometimes raidZ2 backing the system is faster than raid10.
 
danswartz said:
Something sounds broken with the test. There is no way raidz should be faster than raid10.
Click to expand...

It is because ZFS doesnt used fixed block sizes. It uses random block sizes etc... one block or sets of blocks might be 4K and another might be 32K.

It also might have to do with how ZFS uses TONS of RAM for IO before committing to disk and traditional RAID uses raw DISK IO and far less RAM in otherwords your nice LSI megaraid would have come with 32GB of RAM if it actually needed it for traditional hardware raid.

This is all opinion and a little experience of course.
 
FlangeMonkey said:
Thanks for this, its really going to help with my hardware choices. I'm specifically looking at 4 disk and its interesting that RAIDZ outperforms RAID10.

tangoseal are you going to perform similar tests with the WD reds?
Click to expand...

I am yes... right now I am burning in my server. And so far it appears that memtest86 is at 91% completed testing 32GB of Registered ECC DDR3-1600 Kingston memory.

Next I will commit to doing a 4 hour Prime95 and if I have no glitches I will then certify my server as okay to start benchmarking etc... after benchmarking and I find all the weak and strong spots I will then commit to a full production setup. Probably around monday or tuesday I will go hot with real live data and not touch it anymore.

Let me add this..... If you are building a server and NOT using Supermicro you are missing out on what might be one of the best tools in any server EVER...... IPMI... It is amazing. I can power cycle my server and everything using IPMI. I am new to it and I am satisfied extremely with its abilities. I am a traditional APC switched PDU and IP KVM kinda network guy. I've used Dell Drac before but it is not as good as Smicro IPMI imho.
 
tangoseal said:
I am yes... right now I am burning in my server. And so far it appears that memtest86 is at 91% completed testing 32GB of Registered ECC DDR3-1600 Kingston memory.

Next I will commit to doing a 4 hour Prime95 and if I have no glitches I will then certify my server as okay to start benchmarking etc... after benchmarking and I find all the weak and strong spots I will then commit to a full production setup. Probably around monday or tuesday I will go hot with real live data and not touch it anymore.

Let me add this..... If you are building a server and NOT using Supermicro you are missing out on what might be one of the best tools in any server EVER...... IPMI... It is amazing. I can power cycle my server and everything using IPMI. I am new to it and I am satisfied extremely with its abilities. I am a traditional APC switched PDU and IP KVM kinda network guy. I've used Dell Drac before but it is not as good as Smicro IPMI imho.
Click to expand...

I really look forward to seeing your results.

Just a cautionary heads up. I've heard of the WD Reds and the Seagate's failing during the first 2 or 3 week period in batch and if you bought from the same place same time, you don't want multiple failures. I usually expect any disks to fail within the first 2 or 3 weeks and with you having an extensive burn-in, I thought I'd mention it.

For me this is my home build. I'm not using the Supermicro and I'm happy you like the IPMI. My experience hasn't been great, We have several proximity Data Centers, which we used Supermicro's for Storage. The IPMI keeps crapping out and the only fix is a hard boot, soft reboot's did nothing because its Out of Band. Its been better since a firmware update, but it still happens from time to time.
 
Here's the thing, though. WIth raid10, sequential read should be able to hit all N spindles, whereas for raidz, you are only reading N-1 spindles (N-2 for raidz2), Writing I would expect raidz* to be faster, sure. Certainly for random IOPS raid10 will be better. I would have expected raidz* to be slightly (not greatly) slower even on the sequential workload. Without seeing how it's testing it it's hard to say.
 
It depends on how the test runs....

If it's a single large file sequential read, raid10 will only effectively read from N/2 drives, whereas raidz will read from N-1.

If you break it down even further, raid10 will read each recordsize block from a single spindle, whereas raidz will read the same block from multiple spindles, up to N-1!
However, raid10 can conduct parallel block reads on up to N/2 spindles where raidz cannot, which is why the difference isn't as big as single vs multiple spindles.


This is all when reading a single file though - things might change once you start doing multiple large file reads....
 
" If it's a single large file sequential read, raid10 will only effectively read from N/2 drives, whereas raidz will read from N-1."

ZFS load balances reads. I am doing a streaming read from a VMDK:

tank 525G 1.30T 12.2K 75 84.9M 438K
mirror 263G 665G 6.13K 35 42.5M 229K
c13t5000C50041AB0C45d0 - - 285 14 21.4M 229K
c13t5000C50041BC61B5d0 - - 286 15 21.4M 229K
mirror 263G 665G 6.11K 39 42.4M 209K
c13t5000C50041BD3E85d0 - - 299 17 21.6M 210K
c13t5000C50041BD703Dd0 - - 287 16 21.5M 210K
 
I'm wondering if there is an implementation difference here? The OP is testing zfsguru, which is freebsd-based. I am working with Openindiana. How about you, billy_nnn?
 
ZFS on Linux here!!
Did use OpenSolaris for years, but didn't fancy the closed shop from Oracle!


ZFS does indeed load balance reads, but what it doesn't do is read the same data twice, so when it reads a block from a 2 disk mirror vdev, it reads it from one of the disks, not both. It may well read the next block from the other drive in the mirror though!

When reading from a stripe, it will read the first block from the first vdev, the second from the second thrid from third, fourth from fourth, fifth from first....and so on (assuming an 8 drive striped mirror)!

With a single vdev raidz of 9 drives, each block will be read from up to 8 drives.
(for best performance, you wouldn't use an 8 drive raidz)



When you are doing a sequential read of a single large file, an 9 drive raidz will likely always be faster than an 8 drive striped mirror!
However, start doing multiple large file sequential reads and it may be different!

Random I/O is a different proposition again though!
 
" Did use OpenSolaris for years, but didn't fancy the closed shop from Oracle!"

Understood. I felt OI was a different proposition. I tried ZoL for a couple of weeks but switched back when I had two panics doing routine things. It's still a 0.xx rc product, so I can't really use it for a home/soho production environment...
 
danswartz said:
Here's the thing, though. WIth raid10, sequential read should be able to hit all N spindles, whereas for raidz, you are only reading N-1 spindles (N-2 for raidz2), Writing I would expect raidz* to be faster, sure. Certainly for random IOPS raid10 will be better. I would have expected raidz* to be slightly (not greatly) slower even on the sequential workload. Without seeing how it's testing it it's hard to say.
Click to expand...
parity is often faster for sequential workloads.
 
Yeah, I was reading a thread elsewhere where richard elling, bob friesenhan and others were talking about this. From my PoV, random read is the the most important, since the #1 job of this server is ESXi datastore...
 
indeed, mirroring is great for random IO. rebuild math gets a bit sketchy as the pools grow though.
 
@sub.mesa knows how the test are implemented. I only tested sequential read/write...too bad i already filled in quite some data so can't do random IO tests.

although i used ZFSguru for testing, i am currently using Solaris 11.1 now. the pool version remains at 28 so i can easily switch.

Yes i am using a Supermicro board, X9SRA, although the case is Norco 4224. I have never used IPMI though. What is it used for?
 
Remote management. Ethernet access to a virtual vga/keyboard/mouse, virtual CDROM for install media, etc..
 
I tried ZoL for a couple of weeks but switched back when I had two panics doing routine things. It's still a 0.xx rc product, so I can't really use it for a home/soho production environment...

Hmmm, never had any problems like that - though I can see why you'd be put off if you did!
ZoL seems solid enough to me, though I only really use it for media storage now (mostly movies, but a few photos and some music too, as well as backup storage for the various PCs around the place).
No virtualisation here :)

I switched to Linux really because I needed a transcoding capable uPNP/DLNA server, and there didn't seem to be any available on OpenSolaris without compiling it yourself, and I can't be bothered with going down that route again (laziness on my part I admit, but I just don't have the time either really).
I did look at OpenIndiana, but it's pretty much the same there - fine if all you want is a ZFS server, but if you need to press your server into other duties, you just don't have the choice you get with Linux - a shame really, but that's the reality!
Tried a few distros, but settled on Ubuntu in the end (one of the best things about ZFS is that you just export the pool from one OS and import it into another :) ).


As for ZFS itself, you made the right choice in striped mirrors for your usage - you'd likely find the performance of raidz and derivatives to be quite poor in comparison on small file random I/O.
That's not really because of parity itself though - rather the different vdev layout!
 
I don't know of any situation offhand where raidz* will have comparable random read performance to raid10, do you? Even if you have a large number of disks enough to have several vdevs, by definition, raid10 will always have more vdevs for the same number of physical disks, no?
 
danswartz said:
I don't know of any situation offhand where raidz* will have comparable random read performance to raid10, do you?
Click to expand...

Not really, unless you "hit" most things from ARC/L2ARC, but then that's probably not truely random!
This is assuming too of course that you generate enough random read I/O to notice!

Even if you have a large number of disks enough to have several vdevs, by definition, raid10 will always have more vdevs for the same number of physical disks, no?
Click to expand...

Yep...that's about the size of it!! :)

You decide on the type of I/O you'll be doing, the required resilience to drive failure you need and the acceptable storage overhead, and then choose your zpool layout accordingly!
There's no one size fits all, but for general purpose use then striped mirrors is probably the best option overall if you can accept the 50% overhead!
Large files, like movies etc, are a specific use case really, one where raidzX is probably a better option once you get over 5-6 spindles (and are only serving to a few clients) - certainly from an overhead perspective, even if you can't make use of the higher sequential I/O performance (1600MB/s is fine on paper, but how do you actually use it? - serving HD movies only needs about 5MB/s (per movie stream), and few people have a network at home which can deliver 1600MB/s of throughput for file copies (not that many companies do either)
 
I did look at OpenIndiana, but it's pretty much the same there - fine if all you want is a ZFS server, but if you need to press your server into other duties, you just don't have the choice you get with Linux - a shame really, but that's the reality!
Click to expand...

That's precisely why I moved to ZoL. It's about as stable as OI was for me but it's has all of the precompiled packages of Linux which makes it extremely powerful.
 
I'd love to switch back to ZoL, but like I said, two panics in two weeks, neither of which got any resolution from the devs (heck, the second panic report wasn't even acknowledged...) I know they're busy and all, but I've never had a panic or hang under OI in 2 yrs on that appliance. Yeah, for me, it was just being a storage backend. I'd switch to it if I could run a stable, performant hypervisor too, to avoid the extra network traffic for the virtual disks. I tried OI+vbox (slow) and proxmox(ZoL)+vbox (the aforementioned two panics). Maybe it was the proxmox-patched kernel, I dunno, but I went that route because proxmox had kvm+gui already baked in. Otherwise, I guess I'd have to pick a linux distro with kvm and look for a decent gui to manage it (yeah, I know I can do kvm from the CLI - not interested in that :) ) What do you run, kac77? Assuming this is a hypervisor too (no reason to assume it is - just wondering...)
 
If all you need is a storage server, why do you need a hypervisor?

Maybe I'm missing the point here (and I've never used ProxMox myself), but why did you need to use vbox with ProxMox?
 
I cant even download OI.... I was thinking of testing OI + NAPP-IT, currently testing ZFS-Guru,, but the site is not letting me even connect to it. I get some wierd error when hitting openinidiana.org
 
"If all you need is a storage server, why do you need a hypervisor?

Maybe I'm missing the point here (and I've never used ProxMox myself), but why did you need to use vbox with ProxMox? "

Good questions. I lose track sometimes of where I have posted what, so I didn't notice lack of context here. The answer: my previous setup was an all-in-one, with ESXi as the hypervisor, and a virtualized OI exporting storage via NFS/CIFS. I was persuaded by someone that the overhead of having the hypervisor have to shuffles disk I/O through a nic (even a virtualized one with bandwidth in excess of 1gb/sec) was slower than desirable. That if I could run the hypervisor and the ZFS storage on one physical box, guest disk performance would be much better. I initially tried virtualbox on a native OI, but the guest disk perf was not that good (as well as other issues, which may be due to the latest virtualbox major release [4.2]). I had played with proxmox in the past, which is a debian-based hypervisor using kvm and/or openvz containers. So I installed proxmox and then installed ZOL on top of that and set up some KVM guests. Performance was good, but I did get those two panics (which like I said, were not root-caused or resolved...) Does this make more sense? Sorry for the confusion...
 
danswartz said:
What do you run, kac77? Assuming this is a hypervisor too (no reason to assume it is - just wondering...)
Click to expand...

I'm running Ubuntu+ZoL ppa. I haven't experienced a single kernel panic and it's been running for months. I've used KVM for production as well. No problems there. Could Proxmox be creating this problem?

I use KVM at home as well. I should be able to test this later and get back to you.
 
Ah - that explains the Proxmox and ZoL, but I still don't get why you even need a hypervisor in the first place (unless you need to run multiple VMs on the same physical server of course).

What actually panicked - the guest or the hypervisor itself?

One option might be to run a full Linux distribution (eg Ubuntu, Mint, Fedora etc) with ZoL for the main storage server and run Virtualbox or VMware player for any guest OSes you may need, though once running a full Linux OS you may find you can do everything you need on that, without needing any virtualisation! I suppose that depends what you are running though!
 
I run 4 guests in production: a windows xp for my wife (she telecommutes with it), a windows xp for me to telecommute occasionally, a centos pbx (pbx in a flash) and a ubuntu mail/web/icinga/etc server. Both panics were in the kernel ZFS code :( Unfortunately, people have had issues with the latest virtualbox (4.2.x) which is what I tried.
 
kac77 said:
I'm running Ubuntu+ZoL ppa. I haven't experienced a single kernel panic and it's been running for months. I've used KVM for production as well. No problems there. Could Proxmox be creating this problem?

I use KVM at home as well. I should be able to test this later and get back to you.
Click to expand...

I was suspicious of this too - proxmox has quite a few patches against the 2.6 kernel they run (to support their openvz PVs as well as the KVM guests...) I'm in the middle of installing a ubuntu 12.04 and am going to look at xen vs kvm and see how that goes...
 
thinking about building myself a couple small ZFS NAS rigs, what's the good rule of thumb on CPU and memory?

thinking about 8 spindles, probably only 8x 1TB drives... i've heard 1GB per 1TB data, is that a good rule of thumb? i would put 16 in just to be sure... i heard that using registered memory isn't needed for ZFS boxes... true or false?

also i didn't think you needed much CPU power at all, figured the lowest end i3 would do just fine...

these would be small NAS' service NFS or iscsi for VMs to 2 hypervisors, havn't nailed down exactly how i'm going to do the private storage network between the hypervisors and the NAS yet, but i wanted to start w/ the shared storage

was probably just going to grab one of the cheaper 8 port LSI/Intel sas jbod adapters...
 
danswartz said:
I run 4 guests in production: a windows xp for my wife (she telecommutes with it), a windows xp for me to telecommute occasionally, a centos pbx (pbx in a flash) and a ubuntu mail/web/icinga/etc server. Both panics were in the kernel ZFS code Unfortunately, people have had issues with the latest virtualbox (4.2.x) which is what I tried.
Click to expand...


See, this is the bit I just don't get....:confused:

Why do you need to run Virtualbox with Proxmox and ZoL?
 
You're mixing up two of my experiments. I didn't run virtualbox with proxmox. I tried running virtualbox on top of openindiana - that sucked performance wise. I also tried running proxmox with ZOL on the underlying debian (patched) kernel. That performed okay, but was unstable. Is that clearer?
 
goodcooper said:
thinking about building myself a couple small ZFS NAS rigs, what's the good rule of thumb on CPU and memory?

thinking about 8 spindles, probably only 8x 1TB drives... i've heard 1GB per 1TB data, is that a good rule of thumb? i would put 16 in just to be sure... i heard that using registered memory isn't needed for ZFS boxes... true or false?

also i didn't think you needed much CPU power at all, figured the lowest end i3 would do just fine...

these would be small NAS' service NFS or iscsi for VMs to 2 hypervisors, havn't nailed down exactly how i'm going to do the private storage network between the hypervisors and the NAS yet, but i wanted to start w/ the shared storage

was probably just going to grab one of the cheaper 8 port LSI/Intel sas jbod adapters...
Click to expand...

Well zfs will use all the ram it has available for caching. You really dont need a hba unless you want bleeding edge io but you would be surprised at how fast an intel ich10 can be and works perfect with 99% of set ups. However you only have 6 ports on an ich10 controller.

I would go with an ecc supporting amd 965 quadcore and m1015 for the best performance per dollar or an intel i3-3000 series dual core with ich10 and 6 disc for another highly cost effective rig. I just shelled out $2000 for my e5-1620 piweres 32g zfs box.
 
You must log in or register to reply here.
Back
Top