Q: AHCI support driving purchasing decision

[rbm]

I'm going to change my existing mdadm-based RAID6 array to ZFS, probably based around Nexentastor. At the same time, I thought it would be a good opportunity to upgrade my existing case to use hot-swap 3-5 bays so that I could get activity LEDs on the front as well as hot-swap capabilities. Researching further, I may consider a Norco 4116 instead.

My motherboard, an ASUS M2N-SLI Deluxe, does not support AHCI on the 6 on-board SATA ports (just on the auxillary eSATA port). So can someone confirm that the following two statements are true?

(1) If the MB does not support AHCI in BIOS, then getting hot-swap trays or a Norco 4116 case will not provide any significant advantage over using internal bays on my existing case. What I will get from my original list of requirements is activity/fail LEDs only.

(2) If I want hot-swap capabilities, then I would have to get a PCI-X SATA adapter card that supports AHCI and stop using the on-board SATA ports. That, or upgrade my MB to one which supports AHCI.

The answers will help me to plan my HW purchase. For example, if the statements are true, I might go for a Norco RPC-470 case instead.

TIA.
Robert.

 

[sub.mesa]

No hot-swap without AHCI indeed. But why would you want PCI-X? Don't use either PCI or PCI-X, especially for ZFS.

Instead, you may want to buy a new motherboard for example based on AMD SB850 southbridge, which provides 6x SATA/600 natively and fully supports AHCI.

Just curious, is hot-swap that important? For a drive failure that may occur only once a year, is shutting down the box and replacing the HDD then powering on again that cumbersome? I would consider it a nice extra, and if you desire performance using chipset or PCIe storage without any hardware bottlenecks would be of paramount importance.

Not sure how much HDDs that casing can hold, but if you need more than 6 SATA then you could consider a PCI-express HBA, preferably one that is compatible with both FreeBSD and Solaris, the two prime ZFS-capable operating systems with many descendants.

Buying a new motherboard also gives you DDR3 capability, meaning you can buy alot of DDR3 memory very cheaply. I suggest a 4x4GB configuration totalling 16GiB, which should cost you about $160 or so. Prices of DRAM may be on the rise again in second and third quarter. While conventional RAID like mdraid on Linux works fine with few memory, ZFS prefers alot more memory instead. Especially if you want high performance you would need plenty of RAM.

 

[rbm]

Thanks for the reply, sub.mesa

No hot-swap without AHCI indeed. But why would you want PCI-X? Don't use either PCI or PCI-X, especially for ZFS.

Ah, ok. Good information. The MB has PCI, PCI-16X, PCI-8X and PCI-e available. Older technology after all.

Instead, you may want to buy a new motherboard for example based on AMD SB850 southbridge, which provides 6x SATA/600 natively and fully supports AHCI.

There's the thin edge of the wedge. Once I start in with a new MB, then I have to look for a new CPU, DDR3 memory, etc. I've a budget to work within and such expenses blows that budget right out of the water. After all, I've other things like family, mortgage, and living expenses that battle for that same pool of money.

Just curious, is hot-swap that important?

No, hot-swap is not important at all. I felt front panel LEDs and plugability were important. In my current RAID setup, I am having difficulty identifying failed drives (associating the physical drive with the SATA port reported by Ubuntu) so LEDs would help. Also, I am having cabling issues that cause drives to go offline randomly causing RAID rebuilds. I feel I could resolve this issue with pluggable bays; I may be wrong though. Maybe new SATA cables are what I should consider.

ZFS prefers alot more memory instead. Especially if you want high performance you would need plenty of RAM.

So here is where I can put that budget to better use. The machine has 6GB now; 4GB is minimum recommended. I've read that SSD will improve performance by accelerating operations of the ZFS intent log (ZIL). Maybe investing in a 60GB SSD for caching ZILwill be a better investment than a fancy case.

I don't have need for lots of storage capacity; data avilability and integrity is the single driver of choice. So, capacity-wise, 6 drives and 2TB is fine for me for this point in time. I use the server to store important data and some media, and use it to serve media to the family. It's all standard def, not HD, so capacity limits and streaming performance is not an issue for me (READ performance medium). I seldom write to the drive (WRITE performance low). I only ask that the data integrity remain extremely high; it must survive up to 2 symultaneous disk failures thus my choice of RAID6. I will probably choose a mirrored RAID-Z when I convert.

- Robert.

 

[hotzen]

regarding

(READ performance medium) [...] (WRITE performance low)

you neither need more than 8 gig ram, nor an optimized ZIL by using an SSD. AFAIK the SSD-backed ZIL is nice for random access and very high transfer-modes. Better invest in a max-efficient PSU and spend rest of the money for vacation.
And, specificially, you want to use RaidZ2 to survive 2 disk failurs(instead of simple RaidZ/RaidZ1)

 

[sub.mesa]

Thanks for the reply, sub.mesa
Ah, ok. Good information. The MB has PCI, PCI-16X, PCI-8X and PCI-e available. Older technology after all.

I can see no PCI-X on your motherboard. I can see only:

3x PCI 32-bit/33MHz (normal PCI; useless for ZFS)
2x PCI-express x16 (can be used for PCIe x8 cards)
2x PCI-express x1 (can be used for network cards or 2-port controllers)

There's the thin edge of the wedge. Once I start in with a new MB, then I have to look for a new CPU, DDR3 memory, etc.

May not be needed; i thought you had a PCI-X motherboard. But you have a PCIe motherboard instead. With 6GB DDR2 you may not want to upgrade further; DDR2 is more expensive and 6GB is a good value to start with for your first ZFS box.

No, hot-swap is not important at all. I felt front panel LEDs and plugability were important. In my current RAID setup, I am having difficulty identifying failed drives (associating the physical drive with the SATA port reported by Ubuntu) so LEDs would help. Also, I am having cabling issues that cause drives to go offline randomly causing RAID rebuilds. I feel I could resolve this issue with pluggable bays; I may be wrong though. Maybe new SATA cables are what I should consider.

Well spending on hot-swap bays would be an expensive operation. Instead you may want to just take care of your cabling. If you would buy a new LSI controller, you would have less cable clutter by two Mini-SAS cables that can connect up to 8 SATA drives.

My experience is that splitter cables for SATA power are a cause for headaches as well; if these are low-quality then they may stop supplying power to your disks for a moment. Especially if that is multiple disks, this is dangerous. Well, not really for ZFS; you may just have to reboot and things will be back to normal. Still this is not fun, so spending some time on it would be preferable.

I've read that SSD will improve performance by accelerating operations of the ZFS intent log (ZIL). Maybe investing in a 60GB SSD for caching ZILwill be a better investment than a fancy case.

ZIL would accelerate writes and is a dangerous feature. You may want to use L2ARC caching on your SSD instead. This accelerates random reads and is completely safe.

I don't have need for lots of storage capacity; data avilability and integrity is the single driver of choice. So, capacity-wise, 6 drives and 2TB is fine for me for this point in time.

6x 2TB in RAIDZ2 would be preferable, that could sustain two complete failures, or one complete failure with another encountering uBER or unreadable ('bad') sectors on another HDD. 6 disks in RAID-Z2 also falls within the recommended disk count for 4K sector disks such as the Samsung F4 2TB series.

 

[rbm]

Thanks guys! This is really useful informatin I've received. I appologize for the PCI-X confusion; the technology and acronyms are confusing sometimes especially if one does not constantly keep up to date.

I'm off to the store today to spend some cash on improved cabling and possibly the LSI board you suggest sub.mesa. I'll shelve the SSD idea for a while until after I've converted the system to ZFS and run it to gauge performance. And I'll keep the same case/MB combo for the short term with an eye towards a better MB in the long term.

If our storage needs increase in the future, I'll just swap out the 500GG drives I use today for those 2TB 4K sector disks; no need to increase the quantity. I can maintain the same storage configuration (RAIDZ2) as well with this future upgrade.

P.S. SWMBO agrees with you on the vacation, hotzen

Cheers,
Robert.

 

[sub.mesa]

As for controller, make sure you buy a non-RAID controller, like the Intel SASUC8i. This controller may ship in IR mode firmware which enabled RAID, but it can be flashed to IT mode firmware which is non-RAID normal SAS/SATA mode - perfect for ZFS.

So Intel SASUC8i with two mini-SAS cables would a recommended setup. This controller should be useful for quite some time and can be moved to a new ZFS system in the future. The 3Gbps per HDD bandwidth is enough even for much newer and faster HDDs - it'll take awhile before they would hit the SATA/300 bandwidth limits, currently they are scratching the SATA/150 limits so still plenty of headroom.