So recently I have been working on building a second ZFS storage box, and so far it's performance is seriously underwhelming. This is certainly not my first rodeo, there is already a 65TB (usable) storage box encrypted with GELI on the network that performs faster. I am just at a loss for why this new build is so sluggish. Here are the stats:
Gigabyte GA‑870A‑UD3 (1 - 16x PCIe, 1 - 4x PCIe)
AMD Phenom II x4 965 Black Edition (3.4Ghz)
4 x 2GB Corsair XMS3 DDR3 1333mhz
Passive cooled ATI video card in 16x slot
LSI 9211-8i HBA in 4x slot
Norco RPC-4220
Antec TruePower 850w PSU
12 x 2TB Hitachi 0F10311 7200 RPM 32MB Cache
1 x 500GB Seagate SSHD Hybrid for OS
FreeBSD 10.1
8 of the drives are from an old zfs array that was encrypted with GELI, as far as I can tell I have cleared all remnants of metadata from both the previous encryption as well as zfs info. 8 of them are connected to the LSI HBA, and the remaining 4 drives are connected via a reverse breakout cable to 4 of the ports on the motherboard.
It seems that no matter what I do, the performance is always the same. I tried building a zpool with all 12 of the drives, as well as a zpool using just the drive connected to the HBA, the speed is always exactly the same.
Here is what I am getting when benchmarking with dd:
Write:
Read:
As you can see, this is way under what I would expect an array of 12 drives to be capable of, even with the not-so-cutting-edge core system hardware. I have tried a few things so far just to eliminate obvious potential issues - I moved the HBA to the 16x PCIe slot (since it is an 8x card using a 4x slot) and ran headless, in which case the speeds were exactly the same. I was looking at TOP while speed testing and I have never seen the processor exceed 33% utilization.
I intend to do some more testing tonight, such as starting a dd test to some if not all of the drive simultaneously and checking iostat to see what the total throughput is. I also intend to do some firmware flashing, particularly the HBA and possibly the motherboard, but still this should be faster than it is regardless.
Anyone have any other ideas?
*EDIT* - I forgot to mention that this is raidz2 and just to clarify if it was missed the OS is FreeBSD 10.1.
Gigabyte GA‑870A‑UD3 (1 - 16x PCIe, 1 - 4x PCIe)
AMD Phenom II x4 965 Black Edition (3.4Ghz)
4 x 2GB Corsair XMS3 DDR3 1333mhz
Passive cooled ATI video card in 16x slot
LSI 9211-8i HBA in 4x slot
Norco RPC-4220
Antec TruePower 850w PSU
12 x 2TB Hitachi 0F10311 7200 RPM 32MB Cache
1 x 500GB Seagate SSHD Hybrid for OS
FreeBSD 10.1
8 of the drives are from an old zfs array that was encrypted with GELI, as far as I can tell I have cleared all remnants of metadata from both the previous encryption as well as zfs info. 8 of them are connected to the LSI HBA, and the remaining 4 drives are connected via a reverse breakout cable to 4 of the ports on the motherboard.
It seems that no matter what I do, the performance is always the same. I tried building a zpool with all 12 of the drives, as well as a zpool using just the drive connected to the HBA, the speed is always exactly the same.
Here is what I am getting when benchmarking with dd:
Write:
Code:
1024000000 bytes transferred in 6.611905 secs (154872156 bytes/sec)
6.61s real 0.14s user 6.20s sys
Read:
Code:
1024000000 bytes transferred in 2.796616 secs (366156802 bytes/sec)
2.79s real 0.11s user 2.67s sys
As you can see, this is way under what I would expect an array of 12 drives to be capable of, even with the not-so-cutting-edge core system hardware. I have tried a few things so far just to eliminate obvious potential issues - I moved the HBA to the 16x PCIe slot (since it is an 8x card using a 4x slot) and ran headless, in which case the speeds were exactly the same. I was looking at TOP while speed testing and I have never seen the processor exceed 33% utilization.
I intend to do some more testing tonight, such as starting a dd test to some if not all of the drive simultaneously and checking iostat to see what the total throughput is. I also intend to do some firmware flashing, particularly the HBA and possibly the motherboard, but still this should be faster than it is regardless.
Anyone have any other ideas?
*EDIT* - I forgot to mention that this is raidz2 and just to clarify if it was missed the OS is FreeBSD 10.1.
Last edited: