LSI SAS2008-based SAS/SATA-controllers (for ZFS usage)

Hi,

I'm a rather old and silent reader but I'm now in need of some help myself for a change.
I have an old computer running an Opteron 185 and an ATi Xpress 200-chipset which is in need of upgrade since I'm running into some hardware limitations such as not being able to address more than ~3,2Gb of memory even in 64-bit mode and it's running quite hot due to non existing Cool 'n Quiet technology.

Right now I have 4 drives attached to the southbridge (SB450) which performs decently but far from great. I also have 4 additional drives connected to a Promise SATA300 TX4 PCI card that really doesn't like RAIDZ at all (getting 20mbyte/s tops) on FreeBSD 9-CURRENT and it's been like that since the early 7.X-days.

Anyhow, I'm looking at an Intel Q57 or possibly Q67 chipset-based solution due to headless remote access (http://www.realvnc.com/products/viewerplus/hardware.html), much better virtualization support than my current hardware and integrated Intel LAN.

Reading up on controllers the general consensus seems to be that Intel's southbridges performs well and LSI SAS2008 series are suitable as additional controller(s) if you want recent hardware and good performance. Reading regarding the older controllers (LSI 1068) it seems to be preferable to have IR-firmware instead of IT although I'm not really sure what the difference is apart from SAS expander supprt(?).

Doing research of what's available where I live (Sweden) I've found two controllers available to a decent (affordable) price but there seems to be some differences even though the cards appears to be very much alike.

http://www.intel.com/products/server/raid-controllers/RS2WC080/RS2WC080-overview.htm
ftp://public.dhe.ibm.com/common/ssi/ecm/en/xsd03053usen/XSD03053USEN.PDF
http://www.lsi.com/channel/products/hba/sas_sata_hbas/internal/lsisas92118i/index.html <-- Also available, seems to be a mix between IBM (better) but worse than the Intel card looking at features?

From what I can tell which seems odd is that the IBM card doesn't seem to support JBOD (which is what I want?), direct non cached access and RAID5 (needs additional card/key?) as compared to the Intel card although I might be wrong on this.
Looking at IBM's controller there seems to be no firmware upgrades available while Intel at least seems to offer some firmware upgrades. Both cards appears to be clones of http://www.lsi.com/storage_home/pro...pters/sas_hbas/internal/sas9210-8i/index.html with some minor modifications but LSI doesn't seem to offer firmwares for their model. Should I be concerned regarding lack of IR/IT-firmware and which controller would be more suitable? As a side note I've read that both cards have a webinterface, does this work by itself or is software involved in some way? The LSI 9211-8i card doesn't seem to support webinterface, no RAID5(?) although LSI provides firmware upgrades which I guess is better than the IBM card unless you can crossflash it (or any of the cards mentioned).

Best regards,
Daniel

The IBM card is actually closer to the LSI 9240-8i MegaRAID. It does work well with JBOD drives, but you will need a different driver than the other two cards. I got mine working on FreeBSD 8.2 using the mfi driver from LSI's 9240-8i support page, but this required a kernel rebuild. The stock mfi driver included in the kernel does not recognize the card. The major advantage of this card is that it's often available very cheaply as a refurbished or barely-used "pull". I got mine for < $100 USD, compared to as much as $300 for a brand new Intel or LSI 9211-8i.

Unless you can find the IBM card dirt cheap, then I'd say the LSI 9211 and Intel cards are probably closer to what you want for ZFS usage, especially with the IT-mode firmware (you want IT, not IR). They have minimal RAID capability, but since you seem to be using 9.0-CURRENT already, I believe the mps driver in the kernel should recognize them without any additional effort.

If you're getting a new motherboard/CPU anyway, I would strongly recommend looking at a server board, especially if you want headless remote access. Something like the Supermicro X8SI6-F would be perfect for you as it already has an LSI SAS2008 built into the motherboard, as well as dual Intel NICs. The IPMI/baseboard management controller gives you all the network control you could hope for including KVM (even with the system completely powered off) and remote power cycling. You can even mount ISO files remotely and boot from virtual CD/DVD over the network. I did several complete OS installs over the network just yesterday on my X8SIA-F.

EDIT: After re-reading the specs on the Intel card, it also appears to be a 9240-class card like the IBM, but with RAID 5 fully enabled. With the IBM you have to buy a special hardware dongle to enable RAID 5, though for ZFS you really don't need it. So I'd still say that an LSI 9211 or Supermicro X8SI6 cross-flashed with LSI IT firmware should be simpler to live with on FreeBSD 9.0

Hmm,

As far as I can tell both M1015 and the Intel card have the same firmware so I guess both are based on 9240-8i. BIOS revisions are the same (Intel and LSI at least) and you can flash the 9240-8i firmware on the M1015 according to the other M1015 thread I found here.

As for the motherboard it seems to have everything integrated but I have my reservations with everything integrated and the cost would be same as getting a controller card + mobo. Even though I don't plan on of using a GUI anytime soon I'm not sure if I want a board using a Matrox G200(!) since it may need to run Windows etc in the future.

//Danne

I can comment on Solaris support: LSI is pretty solid. I have used a 1068E-based controller from almost every who makes one (Supermicro, LSI/Intel, etc), and they've been great (bar some minor SMART issues).

The 2008-series adds 6Gbps support, in addition to being newer, but support is still quite good.

Regarding IR vs IT, you want to run in IT mode for ZFS usage. You can download the latest firmware from LSI (in my case, it's from Dec. 2010) and drop it on a USB drive with DOS on it, and run a batch file that will happily update your controllers to IT. Currently got 2 Intel SASUC8Is flashed with latest LSI stock firmware, and they are doing great.

I used to run Supermicro (I am selling off my old Supermicro cards, if you are interested...), but I got a good price on the Intel cards on Amazon, so I switched to those (~$140/pc).

Okay, so if possible IT-mode is preferable (which seems to be builtin on 9240 clones). I'm leaning towards the SAS2008-based controllers since they're faster and most likely will have longer support than first gen. By cards, do you mean motherboards or controllers?

//Danne

Double post...

diizzy said:

By cards, do you mean motherboards or controllers?//Danne

Click to expand...

I mean PCI Express controllers, x8 physical/electrical. Supermicro will often integrated one of the controllers onto their board as well (simply dropping the 1068 or 2008 BGA on the motherboard and hardwiring 8 PCIe lanes to it).

So, a little update...

I decided go a somewhat different route which seems work somewhat at least.
I got hold of a M1015 card which is in a DG45ID motherboard by Intel and shows up during boot, listing drives and all. Getting into WebBIOS is a bit tricky since most of the time you'll just get some random beeps before booting the OS. I've flashed the card with the newest LSI firmware available which worked fine and just for fun I tried getting it to work in Windows 7 (as a controller) and here's where the fun begins. It is detected, the LSI driver/Windows says "This device cannot start. (Code 10)", driver versoin is 4.33.0.32. Running the MegaCli utility it doesn't find the adapter at all but if I boot FreeDOS it's found and can be flashed. I'm not sure if it's supposed to work this way but a led named CR1 (top right corner next to the 4-pin connector) flashes on and off every second and I haven't been able to find what it means. Any pointers on where to start or if I'm doing something horribly wrong? All settings are stock on the LSI card and what does the BIOS on/off setting mean?

Best regards,
Daniel

+1 interest, since I just got an M1015 too...
I heard of incompabilities between motherboards and that adapter in the PCIe x16 port, something along BIOS-Size or SMBus-problems. But you seem to be fine since you are at least able to boot with that adapter. That's one adventure I have still to go...

I'm not really sure what the SMBus issue is or how it shows itself but the fact still remains that it doesn't seem to work in Windows at least which would be nice to verify and I still don't know about that blinking LED which occurs on both my cards.

//Danne

Most of the LSI adapters have a blinking "heartbeat" LED somewhere on the board. I believe it's just there to show that the adapter is healthy and the firmware is up and running on the embedded CPU.

Thought I'd give a little update, after getting a bit time to look at this again it looks like my motherboard doesn't like the M1015. I've seen at least one report that the Q45-chipset version of my board (by Intel) behaves the same and also a P67 Asus board. I've tried everything there is in the BIOS (which isn't that much) and also (of course) updating the M1015 BIOS still no go. I don't have high hopes but I'll write a mail to Intel and ask if they can look into it. WebBIOS etc works fine but you cannot load drivers for it, tried with both Win 7 and FreeBSD 8.2.

//Danne

Here is a list of good cards for ZFS:
http://blog.zorinaq.com/?e=10

Thought I would give you the last part of the story...

After flashing ("downgrading" to 9210-8i) this card works very good in the DG45ID motherboard and runs happily with two ASMedia ASM1061 controller cards (2-port SATA 6Gb/s AHCI PCIe Gen 2.0) , an old Intel 100mbit PCI NIC running FreeBSD 9-CURRENT.

//Danne