• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

drive performance for ZFS?

W

wkearney99

Limp Gawd
Joined
Sep 8, 2010
Messages
370
What's going to give me better performance in ZFS, direct access to the drives (through a PCI-x SAT2-MV8) or passed through on a PCIe-8x Areca 1260 card? I've got the choice of using either a pair of the SAT2-MV8 cards or a single 16 port Areca.

Most likely the connected drives will just be SATA3 7200rpm consumer drives, not enterprise. I'll probably only start with 4 drives in a raidz1 pool (possibly 6 in a raidz2).

I'm also wondering about using the 1260 to handle a RAID10 or RAID6 and pass that through to ZFS as a single volume. The idea is to let ZFS handle the compression and caching, while having the Areca card handle the redundancy (it has a 4gb module and the battery). This is for office documents, not transactions, databases or media.

And if I'm passing multiple drives through (not using an on-board RAID) then is it better to put the drives on different controllers? Is ZFS going to be able to get better performance by the drives being on separate controllers, or is the margin so small as to probably not matter much. Especially if there's 16gb RAM for cache and an SSD for the L2ARC (no ZIL planned at this point, SLCs are still expensive).
 
Last edited:
And while I'm asking, what reasonably priced multi-port PCIe (up to x8) cards are worth considering for direct access to SATA drives from ZFS? Running openindiana (solaris).
 
wkearney99 said:
Yes, that's one I've been considering. Any others? Or warnings about ones to avoid?
Click to expand...

that's "the one", period. unless you want to spend $300 for the retail LSI version that comes in a nice box. In the sub <$100 price class, anything that doesn't have a SAS2008 under the hood is certified garbage.

* No offense to OP but I swear if the owners of this forum ever leverage the concept called stickies to give newbies a fighting chance and cut down on duplicate threads, it'll be a miracle.
 
Last edited:
wkearney99 said:
What's going to give me better performance in ZFS, direct access to the drives (through a PCI-x SAT2-MV8) or passed through on a PCIe-8x Areca 1260 card? I've got the choice of using either a pair of the SAT2-MV8 cards or a single 16 port Areca.

Most likely the connected drives will just be SATA3 7200rpm consumer drives, not enterprise. I'll probably only start with 4 drives in a raidz1 pool (possibly 6 in a raidz2).

I'm also wondering about using the 1260 to handle a RAID10 or RAID6 and pass that through to ZFS as a single volume. The idea is to let ZFS handle the compression and caching, while having the Areca card handle the redundancy (it has a 4gb module and the battery). This is for office documents, not transactions, databases or media.

And if I'm passing multiple drives through (not using an on-board RAID) then is it better to put the drives on different controllers? Is ZFS going to be able to get better performance by the drives being on separate controllers, or is the margin so small as to probably not matter much. Especially if there's 16gb RAM for cache and an SSD for the L2ARC (no ZIL planned at this point, SLCs are still expensive).
Click to expand...

on my side:
SAT2-MV8 is pretty minimal on performance :p. M1015 is far away better than SAT2-MV8 on my testing.
SAT2-MV8 does not support hotswap too.

do not use hardware raid card for ZFS as common senses.
if possible to pass through to ZFS VM.
you can use raw mapping or others when no option is available.
you will ease you headaches when using direct pass through.
 
Right, so ZFS is certainly capable of doing good things with direct access to drives. But a lot of it's performance is based on RAM, L2ARC and ZIL options, not necessarily on the final access to the drives. Has anyone tested how the same number of drives performs when connected directly vs through a decently performing hardware RAID card? I'm curious, not arguing one way or the other.

I'm not dealing with high availability here, so not having hotswap is not a deal-breaker. It'd be convenient but not critical for this particular setup.
 
The direct access to drives is mainly so that you can monitor drives and replace them as needed. Even without hot swap, it takes little time to shut down the server, pull a failing drive, replace it, reboot, and instruct zfs to use the new drive instead. If you are using a HW raid controller which hides all of this, you have just neutered all of the checksum protection ZFS provides, since if a drive flakes and provides bad data (undetected by the raid controller), ZFS will not be able to fix this for you, since it will have only one copy of data to operate on. Also, consider that if you go with one or more mirrored vdevs, random read IOPs is better, since ZFS will load balance random reads, whereas many (if not most) HW raid controllers do not (certainly not the cheap ones I've seen.)
 
We agree on the hotswap, it's nice to have but not critical for many simpler situations.

The checksumming aspect certainly makes for a good reason to have direct access and multiple drives, good point to consider.
 
I have found through much testing the MV8 really only performs well on windows, otherwise LSI SAS2008 is my top choice.
 
SAT2-MV8 performance is pretty terrible for zfs imho, cant even saturate gigabit. Stay away from PCI-X for zfs
 
If you are using hardware raid, don't bother with ZFS.
It's pretty much antithetical.

"I'm buying a Corvette Z06 to be used as a grocery-getter driven by my wife." Similar analogy.
Use it as it's intended to be used, or use something else.
 
wkearney99 said:
Well, is it PCI-x or that particular controller?
Click to expand...

This is for general. the chipset limitation.
I can achieve only a half of giga ethernet during reading from nfs, the writing is far worst.
For htpc or movies/musics/videos are ok.
I would against to use as a primary storage that need responses.
 
staticlag said:
SAT2-MV8 performance is pretty terrible for zfs imho, cant even saturate gigabit. Stay away from PCI-X for zfs
Click to expand...

Three or four years ago (prior the LSI 1068 based controllers), the SAT2 MV8 was
quite one of the best cards for ZFS, even used by SUN in their machines.

Today you can use them if you can ignore hot-swap.
I own 8 of them and replace them all now with LSI 1068 or 2008 based ones
- not for performance but hotplug capability (I cannot afford to reboot on every disk replace)

But if the other option is hardware-Raid: use them
 
mwarps said:
If you are using hardware raid, don't bother with ZFS.
It's pretty much antithetical.

"I'm buying a Corvette Z06 to be used as a grocery-getter driven by my wife." Similar analogy.
Use it as it's intended to be used, or use something else.
Click to expand...

You can use hardware-Raid with ZFS like with any other filesystem and keep most of its features but you loose
- money without need
- self repair capability (but you will get informed about errors due to checksums)
- no problem with write whole unless you have battery packs on the hardware-raid-controller
- controller independand pools
 
using hardware raid for zfs is over-killed for me :)
let zfs handle the drives, do NOT let hardware raid control the drives :p.
imagine putting another door between end to end point(ZFS<->drives). something wrong happens, could give headaches to solve the issue.

sat2-mv8 is pretty old. unless the motherboard still support pci-X,
M1015 (LSI chipset) is pretty good in performance including hot-swap.

on my case: I have LGA771 motherboard with 2 PCI-X slots where I put sat2-mv8.
 
Chipset in the SAT2-MV8 card, or the motherboard? I'm assuming you mean the card, as I haven't mentioned the motherboard chipset involved.

Then if it's the card then I'm assuming that spreading the drives across different controllers would reduce some of the bottlenecks, yes?

I'll probably use a 5400-based board instead. That has 4 PCI-e x8 slots vs the 3 PCI-x slots in the current setup. This will also provide VT-d for passing hardware to a VM if needed.

I'm weighing my options. One would be running openindiana native (for fastest ZFS and CIFS) on the machine and then a KVM-based VM for a Windows server or two. The other would be ESXi on the hardware and put both openindiana and windows in their own VMs. The latter requiring more configuration work stitching together the controllers and drives with the appropriate VMs.
 
mwarps said:
If you are using hardware raid, don't bother with ZFS.
It's pretty much antithetical.
Click to expand...
This is not entirely true. With a hardware RAID you're only going to get the benefits the card itself provides. But it presents only a monolithic LUN to the host. In the past this wasn't a bad idea as the host software and performance wasn't nearly as capable. Times have changed and many new options exist. ZFS seems to have developed a much better chance at catching evolving drive failures by detecting corruption, even before the drive itself fails. This versus many hardware devices only detecting outright hardware errors. Since redundancy is pointless without data integrity it would seem unwise to dismiss ZFS in favor of hardware RAID without a lot more thought.

But the point of the thread here isn't to argue for/against ZFS vs hardware RAID, if you want to start that debate, feel free to use the
newthread.gif
button.

"I'm buying a Corvette Z06 to be used as a grocery-getter driven by my wife." Similar analogy. Use it as it's intended to be used, or use something else.
Click to expand...
Weak analogies are less than helpful.
 
I'm not sure I agree with you, wkearney99. Running a ZFS setup with a single HW LUN completely nullifies one of the main reasons for using ZFS. If a drive is going bad, but the raid controller does not know that, ZFS can detect the bad data, but will be unable to repair the error. I have to confess I don't see any legitimate use case for this. So, 'antithetical', I would agree. Not going to comment on the 'corvette for groceries' though :)
 
odditory said:
* No offense to OP but I swear if the owners of this forum ever leverage the concept called stickies to give newbies a fighting chance and cut down on duplicate threads, it'll be a miracle.
Click to expand...

None taken. The trouble with massive 'owners' threads is their bulk. The areca thread is another example. Search certainly helps but, as you point out, eventually some stuff deserves having a sticky post setup.

Eventually it helps to have sub-forums but that's a whole other debate.
 
You must log in or register to reply here.
Back
Top