3ware 9650se behavior with ZFS

[safelyRemove]

I am using FreeBSD 9.1 with a 3ware 9650se and ZFS (raidz2). Latest firmware on the 9650, and latest drivers on FreeBSD.

It has three types of StorSave performance profiles, Protect, Balance, and Performance.

First, when I have drives on the built-in AHCI ports, the drive lights will illuminate in bursts during a write, as expected, and then go out.

On the 3ware, in Performance mode, the lights all come on at the same time, as expected, but roughly 7 seconds after the lights go on with AHCI-ported drives. So, the 3ware is caching. OK.

In Balance mode, the caching doesn't happen, and the lights on both AHCI and 3ware will come on at the same time, BUT, the AHCI drives finish very quickly and go out, while the 3ware drives suddenly begin lighting up one at a time, in proportion to the amount of data written, then they all light up again (serially, for the uber-blocks I assume), and go out. It is much slower, and it is slower than AHCI performance.

There are no in-between settings on the 3ware that I am aware of. I believe when ZFS issues the flush command, the 3ware card begins serializing the writes across *all* ports, rather than letting each drive work in parallel as it should. (The drives are configured as JBOD.) ZFS mirror vdevs have the same behavior.

Can anyone else confirm this, or have ideas to have the 3ware match the AHCI behavior (no 7 second unnecessary caching, but no serializing across ports, either)?

 

[hotcrandel]

I can't confirm this, but i'm using the same card. Had to set up each drive individually as "Single" drives in the RAID BIOS, but ZFS reads the drives individually properly.

During setup I set the cache settings to no read or write cache from the card for each drive.

But it made it a pain as hotswap didn't work cleanly, had to reboot the machine. I may be ditching it in favor of an LSI card.

 

[Nate7311]

Unfortunately, the 3Ware stuff is a more of a pure RAID card, no real HBA or IT mode that you can get with most of the LSI stuff. I've played with 9500s, 9550s, and a 9650 with ZFS, with the most testing done with a 9550 (66Mz/64bit PCI-X) in a repurposed 12 bay Rackable Systems box. The 9650 is essentially the same card with a PCI-E interface. My best results between performance and full functionality was Exporting each drive out as a Single and using the Balanced setting. Even then while they worked, I had some weird random short pauses during any heavy read/write cycles. For my Storage Server at home, I ended up using the hardware raid functionality under FreeNAS as the Raid driver is fully integrated.

 

[safelyRemove]

I can't confirm this, but i'm using the same card. Had to set up each drive individually as "Single" drives in the RAID BIOS, but ZFS reads the drives individually properly.

I'm using the JBOD mode.. I think you press J in the RAID BIOS.

During setup I set the cache settings to no read or write cache from the card for each drive.

I just tried turning off the wrcache, and using Performance mode. That gives the closest to AHCI yet. The AHCI drives finish very rapidly, and the 3ware drives finish quite a bit slower, but in parallel, and without the 7 second delay.

Is there a way to turn on the drive write caches, while leaving the board cache off? I'm also not sure how 'safe' Performance mode is in a crash. Just some data loss, or pool corruption?

But it made it a pain as hotswap didn't work cleanly, had to reboot the machine. I may be ditching it in favor of an LSI card.

I think it works in JBOD mode. Just use the tw_cli /c0/u0 rescan

I had some weird random short pauses during any heavy read/write cycles.

I think turning off rdcache helps on the reads. As for write pauses, that might be what I'm seeing with the serialized writes.