My ZFS update

E

extide

2[H]4U
Joined
Dec 19, 2008
Messages
3,494
PER REQUEST!

So, yeah. Hi guys, I'm James. I have a ZFS array,and it's cool. ...but... well, a few things. First of all, it's full. Secondly, I fucked up the alignment (4k/512E drives) so perf is TERRIBLE.

Ok so yeah, it's based on a 6x2TB RaidZ2 setup. I'm using it a little differently than most. I created a volume that is exported via FiberChannel. (I got FIVE 4Gb FC cards for $20, effin argue with THAT!) So yeah, we have a volume, exported to another PC. That "other pc" is a windows server. NTFS on ZFS over FC. (Used to be iSCSI) Great. I did this because I was not impressed with samba's performance. I am still not impressed with Samba's performance, but will probably skip the whole windows layer this time. I dunno, haven't fully decided yet. WHATEVER

(BTW Once new array is made, I will nuke the old and align it properly..)

SO, My current setup is based on an X58 system with an extra SATA controller built on the mobo, + another 2 port shitty asmedia one. GOOD ENOUGH!

Welp, I got a fairly good sized chunk of extra cash on a recent check, so I wanted to resolve some of the, umm, issues.

I found a pair of SAS2008 cards (LSI 9211-8i/rebranded) for $90 total. YUP
Then I snapped up a Norco 4224 on the egg for (apparently) $100 off. $329 seems like a good price based on the published histories so yeah I got one. Got a few 8087->8087 cables and a reverse breakout cable so I could use 4 of my ICH channels as well.

And then, I ran out of money. I know... such a buzzkill, man.
I DID see some refurb 4TB WD RE's on the egg for $99 each, but they went OOS before I could snag em.

MORE TO COME (I guess..)
 
Uhm, ZFS degrades at 85% and higher fill of a pool if I recall. (Or somewhere around there) You ALWAYS need space

Your crappy performance is due to the entire pool being full not the alignment, although that has some affect not nearly as much as a full pool.
 
ToddW2 said:
Uhm, ZFS degrades at 85% and higher fill of a pool if I recall. (Or somewhere around there) You ALWAYS need space

Your crappy performance is due to the entire pool being full not the alignment, although that has some affect not nearly as much as a full pool.
Click to expand...

Yeah, that and if you fill it 100% full (like zero bytes free) some seriously nasty things happen. Like can't delete files nasty.

I'm starting to get close to the 85% number on one of mine, really need more space but don't have the money at the moment.
 
ToddW2 said:
Uhm, ZFS degrades at 85% and higher fill of a pool if I recall. (Or somewhere around there) You ALWAYS need space

Your crappy performance is due to the entire pool being full not the alignment, although that has some affect not nearly as much as a full pool.
Click to expand...

That may be true, but the crappy perf was there from the beginning.
 
So I got the Norco 4224 and the cables yesterday. Had a bit of an adventure flashing the cards. I have Dell Perc H310's. I assumed I could just follow the same steps as when flashing IBM M1015's. Well, you can't .. quite. Apparently these cards just don't like the IT firmware. I ran into the exact same issues and had to follow the final steps at the bottom of this page under "To Recap:" to flash them: http://www.vladan.fr/flash-dell-perc-h310-with-it-firmware/
The result, I have latest P20 firmware, but IR version. I decided not to flash the BIOS so my system doesnt take forever to boot. (I am booting off a drive on the ICH) If anyone knows how to actually get the IT firmware on the H310's, I would be interested to know.

So, once I got the cards flashed and everything hooked up ... it booted up and all my drives were detected and zfs loaded up everything. My network cards changed name, apparently because I put the NIC in a different slot this time. (So it wasnt right in front of the HBA -- allowing it better cooling) So I had to fix that, but otherwise everything was pretty smooth. Thankfully I always do everything by UUID, and not sba/hba addresses.

I took several pics of the progress, I will post them up later.

I did run into a bit of an issue, my heatsink is too tall for this case. I will have to swap over to my stock intel lga 1366 cooler, which is fairly beefy. I might have to drop down my overclock (sacrelidge on a ZFS server, I know, but it's solid) because it is a fair bit smaller. Hope not.
 
So, the images are below. After this post, I probably won't update this thread in a while. I won't have the cash to buy a bunch of drives for a while, but I am ready to go I have the room for 12 more drives right now. I have the 2 cards plugged into the top 4 backplanes, and then the reverse breakout going to the ICH's plugged into the bottom backplane. I will need to get an expander before I will be able to plug in the 5th backplane.

All images are clickable for full-rez.

Here is the case, quite a beauty:


Bit blurry, but here is a look inside:


Another One:


A look at the backplanes, also note the 3x120MM fan wall, and the OS drive tray on the right, both included by default.


A closer look at the molex connectors:


The SFF-8087 connectors:


The right side:


With the fan wall removed, and cables installed:


Better view of cables:


Powersupply and board installed: (Unfortunately I am going to need to pull it back out to swap the cooler. Argh.)


A closer look at the cables:


A closer look at the cables installed, you can see there is no connection on the 5th backplane:


The power connectors installed, I will need a y splitter to get the 5th backplane as well:


A different angle:


The ram:


Here you can see the front of the machine with it on, and also the new heatsink I will need to install.

You can also see the 7 drives in the top, with the light on, which are my ZFS array & hotspare. There is also a drive installed in the bottom right slot, but you can't see the light because the heatsink is blocking it. I have backplane 1 and 3 on one HBA and 2 and 4 on the other. The hot spare is the 4th one over in the top row, so ther are 3 active drives on each controller, not that it really matters.


A shot insside with it all up and running. HBA's in the best PCIe slots :) ..:


Another inside shot:
 
extide said:
So I got the Norco 4224 and the cables yesterday. Had a bit of an adventure flashing the cards. I have Dell Perc H310's. I assumed I could just follow the same steps as when flashing IBM M1015's. Well, you can't .. quite. Apparently these cards just don't like the IT firmware. I ran into the exact same issues and had to follow the final steps at the bottom of this page under "To Recap:" to flash them: http://www.vladan.fr/flash-dell-perc-h310-with-it-firmware/
The result, I have latest P20 firmware, but IR version. I decided not to flash the BIOS so my system doesnt take forever to boot. (I am booting off a drive on the ICH) If anyone knows how to actually get the IT firmware on the H310's, I would be interested to know.

.....
Click to expand...

try this one https://techmattr.wordpress.com/2013/08/30/sas-hba-crossflashing-or-flashing-to-it-mode/

I did flash with 6Gbfirmware (IT) with this instruction -> http://forums.overclockers.com.au/showthread.php?t=1045376

you NEED older sas2flash to crossflash form IR <> IT since starting with P19(I Think...), they block to flash from IR to IT or vice versa.
 
cantalup said:
try this one https://techmattr.wordpress.com/2013/08/30/sas-hba-crossflashing-or-flashing-to-it-mode/

I did flash with 6Gbfirmware (IT) with this instruction -> http://forums.overclockers.com.au/showthread.php?t=1045376

you NEED older sas2flash to crossflash form IR <> IT since starting with P19(I Think...), they block to flash from IR to IT or vice versa.
Click to expand...

Interesting, so I just probably need to use an older sas2flash .. hrmm, that should be pretty easy, I might mess around with this tonight. Although, it IS running perfect with the P20 IR F/W and no BIOS.

diizzy said:
No ECC memory :-(
Click to expand...

I am surprised you mention this, but skip right over the fact that it's overclocked. Lol. I am not interested in buying another 24GB of ram at the moment. It's fine :)
 
extide said:
Interesting, so I just probably need to use an older sas2flash .. hrmm, that should be pretty easy, I might mess around with this tonight. Although, it IS running perfect with the P20 IR F/W and no BIOS.
Click to expand...

is you H310 default firmware? or you flash to 9211 IR firmware (non megaraid).

on my experience:
megaraid IR firmware can pass seemless-ly unconfigured drive as real jbod
and..
mpt2sas IR firmware seems adding middle layer to jbod


I have M1015, H200, and H310,
H200 flashed to dell 6Gb IT firmware.(mpt2sas driver)
H310 flashed to dell 6Gb IT firmware (mpt2sas driver)
M1015 flashed to lsi 9240 IR firmware (megaraid driver)

9211 or dell 6Gb firmware has 256 queue depth that usefull in when running under esxi or proxmox or baremetal...

9240 firmware has very small queue depth..

the question is on you needs:
if you want all in IT mode or need big queue depth. flash to 9211 IT or dell 6G IT (for H200 or H310, this is on easy to do. you can flash with lsi 9211 IT too).

I still have old sas2flash dos version and sas2flash.uefi. if you need.I can post zip files that included m1015 SBR, H200 SBR, 9240 SBR, 9211SBR...


have fun!!
 
Here is what I am currently running:

james@James-Hive ~ $ lsmod | grep sas
mpt2sas 163840 7
raid_class 16384 1 mpt2sas
scsi_transport_sas 40960 1 mpt2sas

james@James-Hive ~ $ lspci | grep SCSI
02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
03:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)


I flashed the P20 2811ir.bin file


Yes, please post the zip file with all of that. I would prefer to go with plain LSI 9211 IT FW, of the latest ver (P20, currently).
 
You must log in or register to reply here.
Back
Top