Need help upgrading/replacing NAS with bad performance

M

mamruoc

n00b
Joined
Apr 26, 2011
Messages
46
Hello everyone,

I am a NAS/storage n00b and has been reading many interesting threads in this forum, especially the thread about ifs performance and tweaking started by sub.mesa!

I need help with my system. This is a system that was put together in 2005 and performance (in my opinion) very bad. This is a NAS that I have inherited.

I have:
ASUS P5B motherboard
Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
8 GB RAM, but the BIOS reports only 7 GB (?)
2x SiI 3512 (2P) PCI cards
2x SiI 3132 (2P) PCIe x1 cards
1x PCI Nvidia graphic card

root@test:~# lspci
00:1f.2 SATA controller: Intel Corporation 82801HB (ICH8) 4 port SATA AHCI Controller (rev 02)
02:00.0 SATA controller: JMicron Technology Corp. JMB362/JMB363 Serial ATA Controller (rev 03)
02:00.1 IDE interface: JMicron Technology Corp. JMB362/JMB363 Serial ATA Controller (rev 03)
04:00.0 RAID bus controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01)
05:00.0 RAID bus controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01)
07:00.0 VGA compatible controller: nVidia Corporation NV34 [GeForce FX 5200] (rev a1)
07:01.0 Mass storage controller: Silicon Image, Inc. SiI 3512 [SATALink/SATARaid] Serial ATA Controller (rev 01)
07:02.0 Mass storage controller: Silicon Image, Inc. SiI 3512 [SATALink/SATARaid] Serial ATA Controller (rev 01)


My setup is as follow:
I have two disks as a mirrored madam OS disks on the ICH8 controller.
The two OS disks has been partitioned, so 120 GB on each disk has been used to create a mirrored spool.
I have then 6 disks (different brands, production year and size) connected on the ICH8 and the other controllers (unsure which is connected to which controller though) in a raidz2 spool using fuse-zfs implementation running on top of latest Ubuntu LTS.

root@test:~# zpool status
pool: tank
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
disk/by-id/ata-WDC_WD1001FALS-00J7B1_WD-WMATV2281765 ONLINE 0 0 0
disk/by-id/ata-WDC_WD1001FALS-00J7B1_WD-WMATV2212285 ONLINE 0 0 0
disk/by-id/ata-WDC_WD10EADS-00M2B0_WD-WCAV50894380 ONLINE 0 0 0
disk/by-id/ata-WDC_WD10EADS-00M2B0_WD-WCAV50923430 ONLINE 0 0 0
disk/by-id/ata-WDC_WD20EARS-00S8B1_WD-WCAVY1924737 ONLINE 0 0 0
disk/by-id/ata-WDC_WD10EADS-00M2B0_WD-WCAV51233976 ONLINE 0 0 0

errors: No known data errors

pool: tank2
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
tank2 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
disk/by-id/ata-SAMSUNG_SP2504C_S09QJ10Y713260-part2 ONLINE 0 0 0
disk/by-id/ata-SAMSUNG_SP2504C_S09QJ1GL834042-part2 ONLINE 0 0 0


A quick benchmark (dd) shows these result:

write mirror zpool
20971520000 bytes (21 GB) copied, 509.798 s, 41.1 MB/s
20971520000 bytes (21 GB) copied, 481.737 s, 43.5 MB/s
20971520000 bytes (21 GB) copied, 518.034 s, 40.5 MB/s

read mirror zpool
20971520000 bytes (21 GB) copied, 473.059 s, 44.3 MB/s
20971520000 bytes (21 GB) copied, 467.46 s, 44.9 MB/s
20971520000 bytes (21 GB) copied, 488.934 s, 42.9 MB/s

write raidz2 spool
20971520000 bytes (21 GB) copied, 453.623 s, 46.2 MB/s
20971520000 bytes (21 GB) copied, 436.275 s, 48.1 MB/s
20971520000 bytes (21 GB) copied, 439.876 s, 47.7 MB/s

read raidz2 zpool
20971520000 bytes (21 GB) copied, 53.6505 s, 391 MB/s
20971520000 bytes (21 GB) copied, 46.2837 s, 453 MB/s
20971520000 bytes (21 GB) copied, 57.4818 s, 365 MB/s

I need help to get a better NAS.
I was thinking of replacing the 4 SilX SATA controllers with an Intel® RAID Controller SASUC8i and put 8x Seagate ST2000DL003.
ASUS P5B is kinda old mobo, but I have a "decent" SPU and enough RAM, so I guess it is not that bad? The Intel controller will be connected to a PCIe x16 slot (no problems?).
The hard drives a 6Gb/s, but I have read that there is not difference for setups like mine between 6 and 3 Gb/s? And therefore the Intel card should be a good solution...

My question is: will this be a good move, or should I take another route. If another route, what is your suggestion?

I am sorry for the long post, I am a n00b with alot of question, bare with me please :)
 
You might consider moving from ZFS-fuse with linux to a kernel implemented zfs in solaris or bsd, that should offer better performance.

The SASUC8i is probably also a good choice if going this route.
 
s0rce said:
You might consider moving from ZFS-fuse with linux to a kernel implemented zfs in solaris or bsd, that should offer better performance.

The SASUC8i is probably also a good choice if going this route.
Click to expand...

Yeah, I and in between freebsd+zfs or openindiana...
Maybe installing either one and do a performance test would be a better for further help?
 
Be aware that for the time being at least, freebsd's implementation of zfs lags quite a bit behind the solaris varieties.
 
yes, or just solaris express if you don't worry about any legal considerations.
 
Or nexentacore (if you intend to install napp-it for the gui) or nexentastor CE with their own gui.
 
I tried nexenta first (I use ubuntu on my laptop and am used to apt-get, etc) and while I liked it I figured I would try solaris 11 express and for me it was a bit easier to get everything running that I wanted and napp-it runs great.

I played with bsd for a bit but didn't like it and didn't want to learn it.
 
Only gripe about SE11 is there are (as I understand it) no patches, bugfixes, etc... OI is intended to be an open source fork of opensolaris, e.g. there should be fixes and such. I didn't really like the nexenta GUI anyway...
 
danswartz said:
Only gripe about SE11 is there are (as I understand it) no patches, bugfixes, etc... OI is intended to be an open source fork of opensolaris, e.g. there should be fixes and such. I didn't really like the nexenta GUI anyway...
Click to expand...

I didn't really know that there weren't any patches / updates to packages when I installed it. I've had to build stuff from source even when packages are available because they are so old (ie. git).

mind you, I just check the OI repo and git is just as old. oh well. building from source it is.

NexentaStor has a gui but the plain nexenta doesn't and you can then use napp-it.
 
Sorry guys, I need help with my hardware performance first, then I need help selecting the "best" OS for my NAS :p

So, I installed latest OpenIndiana (b148?), did a default fulldisc installation, imported the raidz2 pool created using fuse-zfs and upgraded the pool to current version and ran dd.

The result are:
writing 21 GB to disc approx 240 MB/s
reading 21 GB from disc 1.0 GB/s

I wanted to test a little bit more, so I created 210 GB file and read from it and the results are:
writing - approx. 140 MB/s
reading - 1.0 GB/s

I was alot better than using Ubuntu and fuse-zfs, but still not as good as I would like, but to be honest, I do not know how much is OK...
What are you guys' feeling av this result? This is using multiple different harddrives without any tuning (e.g. turning off TLER)...
 
The writing is definitely too slow. Not sure why though... Not an OI issue I don't think.
 
I have a follow-up question.

On a raidz2 setup, will 3 GB/s be enough bandwidth or will it be satured and I should go for a newer card which supports 6 GB/S?
 
Well, I have a 6-disk raidz2 pool with WD blue drives (640GB), and my reads are about 50% faster than the writes. e.g. writing a 32GB file and reading it back gets 254 MB/sec and 398 MB/sec. You're getting about a 10X difference. I know nothing about zfs fuse performance, but that could be it right there.
 
danswartz said:
Well, I have a 6-disk raidz2 pool with WD blue drives (640GB), and my reads are about 50% faster than the writes. e.g. writing a 32GB file and reading it back gets 254 MB/sec and 398 MB/sec. You're getting about a 10X difference. I know nothing about zfs fuse performance, but that could be it right there.
Click to expand...

First of all, I am ver thankful that you are taking time helping me out.
I have now reinstalled the NAS with OpenIndiana b148. The disks was connected to the Intel ICH8 (2 disks) and 2x Sil 3132 PCIe (4 disks). They all should be on different IRQ, I am a bit unsure which type of bus the ICH8 controllers are connected to the mainboard. The Sil 3132 is PCIe 1x.

I used the same pool as fuse-zfs, imported and upgraded it. I got (too) good results when I dd'ed 21 GB, so I ran dd with filesize of 210 GB.

The results was:
Write: approx 280 MB/s
Read: 1.0 GB/s (this was the same for both filesizes)

So maybe the system was not that bad after all... seems like fuse-zfs slowed the system alot.

What I do not understand is the big difference, and specially the 1GB/s read speed which to me seems to have maxed out somewhere in the computer system...
 
Ok, so I have now installed NexentarStor CE 3.0.5 and ran:
dd id=/dev/zero of=/volumes/test/test bs=2M count=100000

I got approx same reuslt as with fuse-zfs and this worries me:

Write:
209715200000 bytes (210 GB) copied, 3237 seconds, 64.8 MB/s
209715200000 bytes (210 GB) copied, 4089.66 seconds, 51.3 MB/s
209715200000 bytes (210 GB) copied, 3899.13 seconds, 53.8 MB/s

Read:
209715200000 bytes (210 GB) copied, 1069.78 seconds, 196 MB/s
209715200000 bytes (210 GB) copied, 1100.83 seconds, 191 MB/s
209715200000 bytes (210 GB) copied, 1095.04 seconds, 192 MB/s

What do you guys think?
Maybe OpenIndiana b148 is reporting something wrong or better driver support, anything?
 
weird. if anything OI is a newer kernel. and Nexentastor is an Opensolaris kernel (with patches and such I think...)
 
Mm, me too... I guess I'll have to reinstall OpenIndiana and run the test once more. zpool in Nexentastor i v26 and I saw that ARC used close to 6 GB of memory..
 
Oki, did more testing now.
destroyed all the pool, added a cooldown time between each dd (10 sec)

On OpenIndiana zpool v28:
Write:
209715200000 bytes (210 GB) copied, 1323.52 s, 158 MB/s
209715200000 bytes (210 GB) copied, 1341.4 s, 156 MB/s
209715200000 bytes (210 GB) copied, 1357.16 s, 155 MB/s
Read: (bottleneck??)
209715200000 bytes (210 GB) copied, 697.98 s, 300 MB/s
209715200000 bytes (210 GB) copied, 699.415 s, 300 MB/s
209715200000 bytes (210 GB) copied, 697.947 s, 300 MB/s

NexentaStor CE zpool v26:
Write:
209715200000 bytes (210 GB) copied, 2997.18 seconds, 70.0 MB/s
209715200000 bytes (210 GB) copied, 4089.07 seconds, 51.3 MB/s
209715200000 bytes (210 GB) copied, 3959.06 seconds, 53.0 MB/s
Read:
209715200000 bytes (210 GB) copied, 1012.88 seconds, 207 MB/s
209715200000 bytes (210 GB) copied, 1012.78 seconds, 207 MB/s
209715200000 bytes (210 GB) copied, 1048.4 seconds, 200 MB/s
 
Mm, a bit less than I hoped, but I have now ordered the SASUC8I, in the week, we'll if I can get a bit better performance...

The choice of OS was also pretty easy when OpenIndiana performed better than NexentaStor CE.
 
Don't worry about ARC ram usage - if programs and such need RAM, it will give it back...
 
danswartz said:
Don't worry about ARC ram usage - if programs and such need RAM, it will give it back...
Click to expand...

Mm, ok, my point was that I read some other place that he/she saw ARC uses 6.9 GB RAM as max, so I guess I had enough RAM for zfs on Nexenta to perform "better"
 
ARC will only matter once the cache gets hot. e.g. running any read benchmark once, or a read of a file big enough to swap RAM won't really matter, ARC-wise... It's mainly useful if several clients are doing small random reads...
 
danswartz said:
ARC will only matter once the cache gets hot. e.g. running any read benchmark once, or a read of a file big enough to swap RAM won't really matter, ARC-wise... It's mainly useful if several clients are doing small random reads...
Click to expand...

Ok, learned something new today too, thanks!
will be updating once I get the Intel card.
 
Oki, I'm back with update. I had to replace my ASUS motherboard due to boot problem when having the Intel SASUC8i in the PCIe x16 slot, I bought a cheap GIGBAYTE G41M-Combo, regretting already that I did not go for a Supermicro...

Anyways, I get AWFUL performance from the system. Using a ST2000DL003 on the ICH7 SATA controller, I get 4.3 MB/s, one a WD green 5400 RPM 2 TB disk, I get 2.8 GB/s...

A zfs raidz2 over 10 disks (8 connected on the Intel) and two on the SATA slave connectors results in approx 5 MB/s...

How to I debug from here? I am using OpenIndiana and napp-it....

Any help is appreciated.
 
On the Intel card, on one disk pool, after some IRQ fixing in BIOS, I get 26 MB/s, still very bad...
 
root@oi:~# time dd if=/dev/zero of=/backup1/zerofile bs=2M count=10000
10000+0 records in
10000+0 records out
20971520000 bytes (21 GB) copied, 898.786 s, 23.3 MB/s

real 14m58.815s
user 0m0.033s
sys 0m23.804s


Singel disk zfs pool, pretty bad, huh?
 
You must log in or register to reply here.
Back
Top