Inexpensive ZFS or BTRFS build

EDIT: SO, in reviewing the thread it appears that the T20 only has 4 sata ports. Is that true? If so, you will need to get a cheap PCIe SATA card to plug in all drives. Can you verify that there are only 4 ports?

Yes, just 4. I'm getting ready to buy one now. It's basically the last thing I need besides an OS drive. The only reason I haven't bought that yet is because I want to see if I have something laying around the house that would work first.

For PCIe SATA card, I want to order something through Amazon prime if possible to get it quickly. diizzy suggesting this one:

http://www.amazon.com/IO-Crest-Port...=UTF8&qid=1453225423&sr=1-1&keywords=ASM1061&

I was originally thinking I wanted the one with external SATA ports (like this http://www.amazon.com/gp/product/B00UJDLW14/ref=ox_sc_act_title_2?ie=UTF8&psc=1&smid=A2HRXDCK7SIH44) so I could configure my "free" port to be an external one for easy access. But diizzy suggesting I fill the 2 ports on the card anyway with 2 of my disks, and leave the "free" port on the mobo. I can see merits to this, I just trust the ports on the mobo more than the card.

edit: I just went ahead and ordered the suggested one.

The 1TB drives just jumped from $39.99 to $49.99. There's a $5 off on a 500GB bringing it to $34.99 which I'll get if one of these drives is important over say, any old laptop drive that I have laying around? Can I just pull one of those out and use it instead for my OS drive?
 
Last edited:
As an Amazon Associate, HardForum may earn from qualifying purchases.
I own this one:" http://www.amazon.com/IO-Crest-Port...5423&sr=1-1&keywords=ASM1061&tag=hardfocom-20

The lower end marvell controllers have issues with vt-d support, I was able to get it working correctly with the one I chose above though, if possible, stay away from 91xx series and try to use the 92xx controllers.

I don't have a ton of knowledge of the ASM controller though.

Knowledge is knowledge, might help your decision a bit.
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
As I mentioned earlier, mainly if you want to keep backups and/or do something else like download torrents, usenet (you don't want to use ZFS for scratch discs). I'll do more than fine with 1/10th of a drive in terms of OS.
 
OK, I have other PC's that I'll use for that stuff. This is just a dedicated NAS basically. So any old laptop drive is fine then?

Pulled a WD7500BPVT out of an Acer Revo that doesn't get used. Planning to use this for my OS drive. Looks like a WD Scorpio Blue.
 
Last edited:
This means that a Raid-10 of 4 disks has 2 x the write iops and 4 x the read iops of a single disk or a Raid-Z of 4 disks.
That's true, but doesn't it miss the bigger picture here?

Rebuilding 1 vdev of 4x HDs must hit every (used) block on every disk to recalculate parity.

OTOH, 2x mirrored vdevs have completely distinct data, so rebuilding the mirror is a drive clone, not a parity rebuild. Won't ZFS rebuild only the degraded vdev?
 
That's true, but doesn't it miss the bigger picture here?

Rebuilding 1 vdev of 4x HDs must hit every (used) block on every disk to recalculate parity.

OTOH, 2x mirrored vdevs have completely distinct data, so rebuilding the mirror is a drive clone, not a parity rebuild. Won't ZFS rebuild only the degraded vdev?

No, ZFS does not trust unverified data, so a mirror rebuild is not a disk clone but a
read and verify all data what means that iops is importand for resilver time, does not matter
the vdev type. This is also why a resilver with many small files last much longer than
with less large files of the same size and a resilver with less data and files is ultra fast.

btw
There are no mirrored vdevs, vdevs are always striped, I suppose you mean disks
 
Yes, just 4. I'm getting ready to buy one now. It's basically the last thing I need besides an OS drive. The only reason I haven't bought that yet is because I want to see if I have something laying around the house that would work first.

For PCIe SATA card, I want to order something through Amazon prime if possible to get it quickly. diizzy suggesting this one:

http://www.amazon.com/IO-Crest-Port...=UTF8&qid=1453225423&sr=1-1&keywords=ASM1061&

I was originally thinking I wanted the one with external SATA ports (like this http://www.amazon.com/gp/product/B00UJDLW14/ref=ox_sc_act_title_2?ie=UTF8&psc=1&smid=A2HRXDCK7SIH44) so I could configure my "free" port to be an external one for easy access. But diizzy suggesting I fill the 2 ports on the card anyway with 2 of my disks, and leave the "free" port on the mobo. I can see merits to this, I just trust the ports on the mobo more than the card.

edit: I just went ahead and ordered the suggested one.

The 1TB drives just jumped from $39.99 to $49.99. There's a $5 off on a 500GB bringing it to $34.99 which I'll get if one of these drives is important over say, any old laptop drive that I have laying around? Can I just pull one of those out and use it instead for my OS drive?


That PCIe card is fine. Put your os disk on there, and then all 4 of the 2TB drives on the integrated -- integrated Intel ports are pretty much always better than external ones, especially low end ones like ASMedia ones.


Yeah just use an old laptop drive for the OS. It isn't super important. Infact, if you want to be spiffy, you can get 2 old laptop HD's and mirror them using mdadm (If using Linux/ZoL) or the appropriate tools on FreeBSD (gdm, I think) but I would suggest Linux/ZoL.


Also, replying to the post above, when I said mirror vdev, I mean 2 disks, in a mirror, in a vdev.
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
@ facesnorth
Your laptop HDD will do just fine.

@ extide
The PCIe card isn't bootable so no go, hence two 2TB HDDs on the SATA card and the other two on the ASMedia card. Your laptop HDD will do just fine. You can either use gmirror or use zfs mirroring in FreeBSD, I'd advice you to use gmirror for now however.
 
Are we coming to a consensus of what the best type of RAID to build now would be for my system? Two mirror vdev's in a single zpool?
 
I'd make it two separate arrays unless you're going for RAID-Z.

Code:
zpool create array0 mirror ada1 ada2
zpool create array1 mirror ada3 ada4
 
No, ZFS does not trust unverified data, so a mirror rebuild is not a disk clone but a
read and verify all data what means that iops is importand for resilver time, does not matter
the vdev type. This is also why a resilver with many small files last much longer than
with less large files of the same size and a resilver with less data and files is ultra fast.

btw
There are no mirrored vdevs, vdevs are always striped, I suppose you mean disks
Communication fail. Let me clarify what I was trying to say. Compare these 2 configs:
Code:
1) TankZ:
- raidz1-0
--- da1
--- da2
--- da3
--- da4

2) Tank10:
- mirror-0:
--- da1
--- da2
- mirror-1:
--- da3
--- da4
Clear?

Now suppose that da1 fails. Rebuilding TankZ must thrash all HDs to recalculate da1's data from parity. OTOH, to rebuild Tank10, ZFS processes only mirror-0. And - while it checksums everything to verify integrity - new da1's data is cloned from da2, not computed from parity across da2-4.

If I am correct, rebuilding Tank10 does not actually touch ALL data; mirror-1 remains independent. I agree that IOPs still matter (because ZFS must walk up metadata tree), but vdev type does seem to affect how the rebuild works.
 
@ extide
The PCIe card isn't bootable so no go, hence two 2TB HDDs on the SATA card and the other two on the ASMedia card. Your laptop HDD will do just fine. You can either use gmirror or use zfs mirroring in FreeBSD, I'd advice you to use gmirror for now however.

You Sure about that? Even so I would still put 3 of the drives on the intel SATA ports.

I'd make it two separate arrays unless you're going for RAID-Z.

Code:
zpool create array0 mirror ada1 ada2
zpool create array1 mirror ada3 ada4
Why? Why have two volumes to manage and lose half your speed? Pretty much pointless.

zpool create array mirror ada1 ada2 mirror ada3 ada4

Much better
 
Communication fail. Let me clarify what I was trying to say. Compare these 2 configs:
Code:
1) TankZ:
- raidz1-0
--- da1
--- da2
--- da3
--- da4

2) Tank10:
- mirror-0:
--- da1
--- da2
- mirror-1:
--- da3
--- da4
Clear?

Now suppose that da1 fails. Rebuilding TankZ must thrash all HDs to recalculate da1's data from parity. OTOH, to rebuild Tank10, ZFS processes only mirror-0. And - while it checksums everything to verify integrity - new da1's data is cloned from da2, not computed from parity across da2-4.

If I am correct, rebuilding Tank10 does not actually touch ALL data; mirror-1 remains independent. I agree that IOPs still matter (because ZFS must walk up metadata tree), but vdev type does seem to affect how the rebuild works.


The article from Oracle (https://blogs.oracle.com/roch/entry/sequential_resilvering) descibe what is happening on a resilver:
The .. algorithm was actually going through all of the blocks file by file, snapshot by snapshot. When it found an element to resilver, it would issue the IOPS necessary for that operation. Because of the nature of ZFS, the populating of those blocks didn't lead to a sequential workload on the resilvering disks.

So ZFS reads all metadate for all files spread over the whole raid-10. ZFS is not cloning the missing mirror disk from the good one (like some hardware raid would do) but processes all metadata, calculates checksums from data spread over mirror-0 and mirror-1 and copies data from redundancy (only this is on a mirrored disk) to the resilvered disk when needed. This is why pool iops affects resilvering time.
 
@ extide
Yeah, I'm quite sure about that.

Since mirroring seems to be what he's aiming for I'd do separate mirrors for the sake of simplicity (compared to RAID 10).
 
The article from Oracle (https://blogs.oracle.com/roch/entry/sequential_resilvering) descibe what is happening on a resilver:
The .. algorithm was actually going through all of the blocks file by file, snapshot by snapshot. When it found an element to resilver, it would issue the IOPS necessary for that operation. Because of the nature of ZFS, the populating of those blocks didn't lead to a sequential workload on the resilvering disks.

So ZFS reads all metadate for all files spread over the whole raid-10. ZFS is not cloning the missing mirror disk from the good one (like some hardware raid would do) but processes all metadata, calculates checksums from data spread over mirror-0 and mirror-1 and copies data from redundancy (only this is on a mirrored disk) to the resilvered disk when needed. This is why pool iops affects resilvering time.
Outstanding info. Thanks again.
 
RAM arrived today. PCIe card arrived yesterday. I should be ready to go tonight when I get home. I'm sure I'll need some guidance with setting up BSD and also then setting up ZFS.
 
The non-ECC RAM that came with the T20 was located in slot 2. Any reason for this? After removing this, should I put my 8GB Crucial ECC stick into slot 2 (white) or slot 1 (white)?
 
One major bummer I just caught - the PSU supplies power to the mobo and the hard drive supplies power to the HD's, but only provides 4 SATA power cables. I don't think I have any splitters laying around, but I'll check. That's a bummer because it means I won't be able to get my build running tonight or even this weekend.

Also, am I connecting the OS drive to SATA0? Did we decide 3 of the storage hard drives were best to go straight to the mobo connectors and just use 1 connector on the PCIe card for the 4th drive?

edit: just checked....tons of molex to 2 SATA adapters, but no SATA to 2 SATA splitters...
 
Last edited:
* Put it in slot #1
* Put the OS drive on SATA 0
* Connect it as you want but I personally would go for using 2 slots on the PCIe card and leaving SATA 1 empty.

Really sure that it didn't come with ECC memory?

In worst case you can already prepare the OS. :)
You probably want to use Samba 4.3 etc too.
 
I looked up the model on the RAM it came with - M391B5173QHO. It's def non-ECC.

I guess I'll boot it up with just the OS drive and 3 of the storage drives. I can set up BSD, just not ZFS until I have all 4 storage drives powered up I guess.
 
Those $99 4tb hard drives are back in stock at newegg FYI. I'd consider getting a couple, but how would that affect my build? I'd need a different case or enclosure, a different PCIe card. And how would it affect my zfs pool options having drives of different sizes?
 
That depends on what you're planning.

If you just want to add another HDD that's fine since you have one spare SATA connector but you have no physical bay for it unless it's a 2.5" (which I doubt being 4TB). If you add and/or replace a HDD to an existing array it'll just as much space as the other drives the rest would be "wasted".

If you want to add another array you have a few options...

Option 1# You need an additional controller card (HBA) like the IBM M1015, a new (larger) case and a more powerful PSU. Keep in mind that these controller cards are picky about drives and "Green"-branded drives in general work poorly if at all.

Option 2# Get an external "storage" case and use the port multiplier ability of the ASMedia card. This is a cheap way of getting more HDDs connected but you'll share bandwidth so it's not going to be very fast and I'm not sure how well it works in real life. FreeBSD does support it but the driver author expressed some concerns about how error handling is being done by controllers when using the PM feature in general.

Option #3 External storage case using a HBA, this will work just like internal HDDs but without sharing the bandwidth. Unfortunately this solution is usually more expensive than getting an internal card, new case and a new PSU. Perhaps other forum members and chip in on some good "deals" here.
 
Any suggestions on case, PSU, card deals that would allow expansion? I haven't built my array yet. Before doing so I'm just briefly exploring what would be involved financially.

What would be the best type of array to create to achieve 8tb useable space with the ability to lose 2 disks and still utilizing the 4 2tb drives I currently possess? Same question but with 10-12TB usable space?

Thanks!
 
"best use" is using datadisks as a power of two, so prefer
a pool with one raid-z2 vdev of 6 x 2 TB disks = 8 TB useable

a pool with one raid-z2 vdev of 10 x 2 TB disks = 16 TB useable
or a pool with two raid-z2 vdevs of 6 x 2 TB disks each (12 disks) = 16 TB useable but double iops
 
Yes you can on different vdevs.
I do this with my backup systems where I started with several vdevs of 2 TB disks. When I needed more capacity I began to replace some 2 TB vdevs with 4 TB disks to increase capacizy.

Only problem. With ZFS iops scale usually with number of vdevs and sequential performance with number of datadisks. In such unbalanced pools iops may be like a single disk and sequential performance like a single vdev as data may be placed only on one vdev.
 
Case: Fractal Design XL
https://www.google.com/shopping/pro...d=0ahUKEwjB9qf0-c3KAhUJWSwKHbLKDyAQ8wIIiwUwAQ

https://www.google.com/shopping/pro...d=0ahUKEwjB9qf0-c3KAhUJWSwKHbLKDyAQ8wIIgwUwAA
Great price over at Provantage right now (about 110$)

HBA: IBM M1015 is a safe bet, around 110$ or so (avoid buying these from China).

Cabling: Get LSI, Adaptec branded cables or any other major band.
~15-20$ each

PSU: 500-600W will be more than enough as long as you get a decent one however right now they're just as expensive as ones in the 700-800W range.

https://www.google.com/shopping/pro...d=0ahUKEwifx_3r-s3KAhWDliwKHQkmBIcQ8wIIyAQwAA

https://www.google.com/shopping/pro...d=0ahUKEwjV9eD0-s3KAhWIDCwKHbhLAB4Q8wIIngYwAQ

https://www.google.com/shopping/pro...LBSwKHd62CW4Q2SsIEg&ei=PM2qVsSsKYuLsAHe7abwBg

You also need this adapter if you want to use a regular PSU with that mobo...
http://de.aliexpress.com/item/ATX-2...oard-8Pin-Male-Adapter-Power/32346420024.html

FWIW, mine runs along fine using a Corsair HX 520W PSU and 16+ HDDs =)
 
Last edited:
Hmm...

Having done a bit more research this seems like a more suitable solution pennywise.

Grab a LSI 9200-8e controller
http://www.unixplus.com/products/lsi-sata-sas-host-bus-adapter
http://www.ebay.com/itm/LS-9200-8e-...446938?hash=item5b1a7d5b5a:g:hCsAAOSwnOJWFBod
http://www.esaitech.com/lsi-logic-corporation-lsi00188-sas-sata-9200-8e-6gb-s-hba.html
http://www.ebay.com/sch/i.html?_fro...0.TRC0.H0.X9200-8e.TRS0&_nkw=9200-8e&_sacat=0
I'd advice you to avoid sellers from China.

Grab 2 of these:
http://www.addonics.com/products/ad4smsas.php
AD4SMSAS-PCI

Cables... (2x)
http://www.ebay.com/itm/NEW-Tyco-1-...588287?hash=item56798ec97f:g:ME0AAOSwjVVVw-Qj

PSU (if needed, overkill but smaller ones are more expensive):
http://www.newegg.com/Product/Product.aspx?Item=N82E16817151146

Reuse old PC case (any will do) or grab a cheap new one
Tell the PSU to be always on or use a small switch
http://www.instructables.com/id/How-to-power-up-an-ATX-Power-Supply-without-a-PC/
http://www.amibay.com/showthread.php?41226-How-to-Modify-ATX-PSU-to-be-always-on

Voliá, 8 more HDDs :)

Would be nice if anyone else could confirm that I didn't messed up ;-)
 
Last edited:
As an eBay Associate, HardForum may earn from qualifying purchases.
Very interesting. So for around $175 + a case to hold up to 8 drives if I can't find one laying around + the cost of drives I can add up to 8 drives to my existing system. So I could start with 2 or 4 for example. And if I wanted to add 4-6 more later, I could do that, but would have to set them up as a new zpool.

Say I just add 2 4tb drives right now. So I'd have 4 2tb and 2 4tb drives. What would be my best options at that point for setting up my zfs pool?
 
4x2TB --> raid-z (RAID-5)
2x4TB --> mirror (RAID-1)

You'd also of course need SATA cables from the AD4SMSAS-PCI card to the HDDs themselves. I didn't add these as you (as in ppl here) have such laying around at home already.
 
Last edited:
I looked up the model on the RAM it came with - M391B5173QHO. It's def non-ECC.

The single google hit with screenshots on ebay show they are exactly Unbuffered ECC RAM. Count the chips, there should be 9 identical ones on the populated sides.
It makes no sense to build with a motherboard and CPU that support ECC and then save on the RAM.
 
The single google hit with screenshots on ebay show they are exactly Unbuffered ECC RAM. Count the chips, there should be 9 identical ones on the populated sides.
It makes no sense to build with a motherboard and CPU that support ECC and then save on the RAM.

Oh my bad, I think the one I looked at was a result that came up with a different model # than what I typed in.

So it actually makes sense to stick this 4GB stick back into the PC into slot 2 correct?
 
4x2TB --> raid-z (RAID-5)
2x4TB --> mirror (RAID-1)

You'd also of course need SATA cables from the AD4SMSAS-PCI card to the HDDs themselves. I didn't add these as you (as in ppl here) have such laying around at home already.

SATA cables aren't a problem.

Are there other good options for my RAID pool setup? For some reason this seems like a wonky setup to me, but there's a lot I don't yet understand.

Some quick searching came up with this:

RAIDZ1 vdevs should have 3, 5, or 9 drives in each vdev (IDEALLY 5)
RAIDZ2 vdevs should have 4, 6, or 10 drives in each vdev
RAIDZ3 vdevs should have 5, 7, or 11 drives in each vdev

I'm also not sure on the pros/cons of mixing drive sizes.
 
So it actually makes sense to stick this 4GB stick back into the PC into slot 2 correct?

Yes. Modules can be placed very flexibly since Core2 at least. Servers usually have specific requirements but the T20 and likes are more desktop.

For T20:
channel A: memory sockets 1 and 3
channel B: memory sockets 2 and 4

You'd want to load up the channels equally, so you'd fill slot 1 then 2, 3 and 4.
If the channels are not equally populated it should revert to Flex Memory (mix of dual and single channel).
 
edit: just checked....tons of molex to 2 SATA adapters, but no SATA to 2 SATA splitters...

I resorted to stripping some insulation and soldering a cable with 2 SATA power connectors to the single power cable they supplied.

You can get some nice SATA splitters (InLine 29683V, but only available in western EU).

Another point I'd like to mention is that the PEXSAT32 was a total failure for me. It either did not show up on the PCI bus or does not list the SSD-s on it.
I was planning to play around with block level caching under Linux, I'll need another solution for this.
 
Last edited:
I resorted to striping some insulation and soldering a cable with 2 SATA power connectors to the single power cable they supplied.

You can get some nice SATA splitters (InLine 29683V, but only available in western EU).

Another point I'd like to mention is that the PEXSAT32 was a total failure for me. It either did not show up on the PCI bus or does not list the SSD-s on it.
I was planning to play around with block level caching under Linux, I'll need another solution for this.

I considered hacking a cable together, but I'm not really in a rush. I ended up ordering this http://www.amazon.com/gp/product/B00V6QA65G?psc=1&redirect=true&ref_=oh_aui_detailpage_o02_s00 and it worked fine. I have 2 spares now.

http://www.amazon.com/gp/product/B005B0A6ZS?psc=1&redirect=true&ref_=oh_aui_detailpage_o05_s00 this is the card I bought. It works fine. I just don't like that it seems to have taken over ada0. I have my OS drive plugged into SATA0 on the mobo, and when I disconnect the drive connected to the PCIe card, it shows my OS drive as ada0, and the 3 storage drives as ada1-3. But when I connect the drive to this PCIe card and boot up, that one shows up as ada0, and everything else gets shifted down.
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
But when I connect the drive to this PCIe card and boot up, that one shows up as ada0, and everything else gets shifted down.

Quite normal with this controller as it uses the same driver as the mobo's and it probably gets scanned sooner. Consider yourself lucky, at least it works :). I'll hunt around for an ASMedia one. Thanks for the tip!
 
@ facesnorth
Regarding the ada0 renumbering that's fixable =)
Can you post a full dmesg otherwise I have no idea what they're named on your system.

@ carobee
Marvell controllers are pretty crappy which I quite specificly told facesnorth that ASMedia is the way to go if you want/need cheap SATA controllers that just works (tm). :)
Where in EU are you?
 
Back
Top