ZFS: vdevs across HBAs

C

choatie88

Limp Gawd
Joined
Oct 31, 2001
Messages
462
I'm thinking of building a storage server with a number of HBAs. Does anyone have experience on connecting all disks in a vdev to 1 HBA vs. connecting each disk in a vdev to a different HBA?

If you had each disk on a different HBA, you gain some protection against failure of the HBA. But I wonder if there is a performance penalty since you now have to talk to each HBA to read a stripe off the vdev...
 
There should be no performance penalty - generally speaking, in many cases it will be faster to use multiple HBAs.
However, it depends on data layout and access patterns, plus how you get data to and from the device (if it's a NAS on gigabit for instance, it all becomes a moot point as the network becomes the performance bottleneck).
It also depends on how fast the HDDs are, how fast the cards and busses are, how many HDDs there are....and so on - in some cases there will be little real world difference.

Each I/O means talking to a HBA - if it's a single HBA and you do 4 I/Os for instance, you'll talk to that HBA 4 times, and queue 4 I/Os there.
With 4 HBAs, you'd talk to each HBA once and queue one I/O in each HBA!
 
For instance, the Sun X4500 thumper had 48 disks. And 6 HBAs. So you could do a big zpool out of several raidz1. Each raidz1 consisted of six disks. Thus, the zpool consisted of eight raidz1 having six disks each.

Each disk was connected to a different HBA. If one HBA crashed, the entire zpool was still up and running becase each raidz1 had lost only one disk.

Good practice.
 
Thanks for the input. I guess the disks are so much slower than the PCIe bus that the latency of talking to each HBA is a non-issue.

Does the situation look any different for large contiguous reads?
 
For sequential access no, that would unlikely ever max out the iop max count for an hba.

For random access, and enough disks (or a few ssds) you could max out the iop max for an hba.
 
You must log in or register to reply here.
Back
Top