need help with ZFS build

D

doublejj0305

n00b
Joined
Oct 25, 2009
Messages
48
Here is the equipment I have so far for it:

GIGABYTE GA-H55-USB3 LGA 1156 Intel H55 HDMI USB 3.0 ATX Intel Motherboard
Intel Core i3-530 Clarkdale 2.93GHz LGA 1156 73W Dual-Core
CORSAIR HX Series CMPSU-650HX 650W ATX12V v2.2
G.SKILL Ripjaws Series 4GB (2 x 2GB) 240-Pin DDR31600
Norco 4220
Supermicro AOC-SASLP-MV8 8-Port SAS SATA Add-on Card

HDD:
2x 1.5tb 7200 barracuda's
2x 2tb WD EADS
1x 2tbWD EARS
Considering purchasing 4x 2TB Samsung F4's

I have never used ZFS before and doing some research. The only purpose of this machine is to serve HD content to three HTPC's around the house Here are some questions:

1) Is this hardware sufficient for ZFS and will it play nice with Opensolaris or FreeBSD?
This is the equipment I currently have. I would have liked to have gone Supermicro board with ECC RAM, although this is what I purchased before even knowing about ZFS. Is there any concerns with not running ECC ram with what I want to do?

2) Not sure which OS I would like to use, I have no experience with either Opensolaris or FreeBSD and have no experience with line commands, although will learn if FreeBSD is the best route. As mentioned up top I only need this to serve HD content, nothing else. I'm coming from a WHS machine and I'm not impressed with "VAIL" specs.

3) I already have a AOC-SASLP-MV8 card so with the board SATA connections that gives me 12 physical HDD connections or 3 x raidz's. This is enough for storage as of now with the HDD's I have at the moment but I'm adding to my storage all the time. Would it be better to purchase a different card now than have to deal with it in the future? In other words should I get something to connect all 20 HDD cages in the Norco 4220?



Thanks in advance for some input. I've read multiple threads about ZFS. but just want to make sure I have a grasp on it before moving away from WHS

J.J.
 
1) Yes 4GB and 64-bit dualcore is good enough for ZFS server. But your controller uses Marvell chip which may give you issues. If it does, you may want to invest in LSI 1068E controller instead; like Supermicro USAS-L8i or Intel SASUC8i.

2) I recommend FreeBSD; it may well be the future of ZFS. It's getting the most complete ZFS version of all OS; 28 + freebsd specific patches. So on the long run, FreeBSD would be a safe choice for ZFS users.

3) As said, possibly that card is unusable due to all driver/firmware issues that plague it. I recommend LSI 1068E and you can combine onboard + several controllers without any problem.

There is one performance issue: if you use RAID-Z with a 'fake 4K drive' then you could have write performance issues. The Samsung F4 EG falls into that category i believe; 4096 byte sectors but 512 byte emulation; just like EARS. That means you will have performance problems on RAID-Z in some cases:

4-disk RAID-Z with 128K recordsize
disk1: 43K
disk2: 43K
disk3: 43K
disk4: 43K (parity)

This is no good combination; 128 / 3 = weird number and not aligned to 4K

This example is better:

4-disk RAID-Z with 192KiB recordsize
disk1: 64K
disk2: 64K
disk3: 64K
disk4: 64K (parity)

Or a RAID-Z2 with 192KiB recordsize and an even number of disks:

8-disk RAID-Z2 with 192K recordsize:
disk1: 32K
disk2: 32K
disk3: 32K
disk4: 32K
disk5: 32K
disk6: 32K
disk7: 32K (parity)
disk8: 32K (parity)

So as you can see, tuning of recordsize may be needed for 'fake 4K' disks in RAID-Z configuration. This does NOT count for mirrors and striping vdevs; only RAID-Z due to variable blocksize thing; RAID-Z behaves pretty much like sector-managed RAID-3 would: it makes all I/O into one-phase I/O write to all disks. Never will you have to do a read+modify+write on RAID3 or RAID-Z.

Once we get 'real 4K disks' then this tuning will not be required any more.

Note that regardless of above issues, RAID-Z give you less IOps performance than mirroring or striping; but sequential I/O should still be good. The more vdevs you have, the better, so 3 vdevs each a RAID-Z could be a nice setup.

Do some benchmarks and testing before you take this in actual usage. Especially, simulating a disk crash (pull the cable on HDD) and how to recover from that is highly recommended. If you do that now, you know your way around when it actually happens. You may wish to document all your findings in a .txt file as a cheat sheet; i often find that useful.

Cheers,
-sub
 
Thanks for the quick reply sub.mesa! I've been looking over the installation of freeBSD on your site and it is very overwhelming, good but overwhelming. I'm not sure if I feel comfortable taking that leap... I will continue to read and re-read
 
Then why don't you try my Mesa .iso image to play with? It comes with everything preinstalled and a web-interface. It's not stable/mature yet so don't use for real; but you could in the future (might take few months to stabilize). You can already try it inside Virtualbox with some fake disks to test out its features and design.

It should work comparable to FreeNAS; you may also want to look at that project, especially since its stable while my project isn't yet.
 
I saw that on your website. I will put it in a virtualbox and see how it goes. Again thanks for the help and direction!
 
On my gigabyte motherboard I have the following PCI slots:
1x PCI Express 2.0x16
3x PCI Express x1
3x PCI


I'm a little confused on if this board will hold 2x USAS-L8i and is there any limitations?
 
Last edited:
At the very most without modification, you'll be able to fit in one USAS-L8i card in the PCI-E x16 slot.

If you're willing to void the warranty, you can fit in two USAS-L8i cards by "opening" up one of the PCI-E x1 slots by modding it:
http://tkdan.com/SC420/
http://www.overclock.net/overclock-net-folding-home-team/439190-great-news-pci-e-x1-folding.html

Otherwise, you might want to look another mobo with two PCI-E x16 slots if you want to use two USAS-L8i cards. Or look up the HP SAS Expander thread here in the forums.
 
any ideas of a good board that I can still use my i3 530 and ripjaw ram that has more PCI-E slots?

Thanks for the reply!
 
The SuperMicro USAS-L8i and other similar controllers have PCI-express x8 interface. This fits in a PCI-e x16 slot, but this card only uses PCI-express generation 1 meaning 250MB/s per lane. 8 lanes makes that a 2GB/s full-duplex bandwidth.

Since you only have one PCI-express x16 slot, you can only really use one controller. Perhaps you can look at LSI 1068E controllers capable of more than 8 disks like 16; but these are quite expensive and many have different chips i think.

What you can also do, but would be less good option:
- 1 SuperMicro PCIe x8 controller
- 2/3 SATA 2-port controllers (JMB363/SiliconImage/Marvell) providing 4 or 6 extra SATA/300 ports total. They share bandwidth with 250MB/s so 125MB/s per device; unless you pick Marvell which shares two SATA/600 with 500MB/s (PCIe x1 generation 2).

PCI-express bandwidths:
<nr_lanes> = generation 1 bandwidth | generation 2 bandwidth
1 lane = 250MB/s | 500MB/s
2 lane = 500MB/s | 1000MB/s
4 lane = 1000MB/s | 2000MB/s
8 lane = 2000MB/s | 4000MB/s
16 lane = 4000MB/s | 8000MB/s
 
Oh yes expander would be an option here, as Danny Bui says. Then you just have one controller on your PCI-express slot and use one additional expander. As long as there's no problems with compatibility, this should work and give enough speeds for HDDs.
 
When i first saw this i thought it was different form factor; until the text lead me to believe they horizontally flipped the image. :D

Anyway looks good, onboard graphics for intel cpus that support it and plenty PCIe.
 
Well I'm thinking of starting from scratch with motherboard, ram, and CPU. The reason being is can just use my old gigabyte setup as a HTPC. I would like to do this right.

So as of now the only thing I have is the Norco 4220 and 650w PSU
What motherboard? supermicro?
CPU?
ECC ram?
still go with 2x USAS-L8i?

Sorry for the change up :(
 
I recommend this base setup:
$115 - Intel Core i3-530 CPU
$190 - Supermicro MBD-X8SIL-F-O Intel 3420mATX Motherboard
$105 - Kingston 2 x 2GB ECC Unbuffered DDR3 1333 RAM
---
Total: $410 plus tax and shipping.
 
Alright, about to get the following:
- Intel Core i3-530 CPU
- Supermicro MBD-X8SIL-F-O Intel 3420mATX Motherboard
- 2x Kingston 2 x 2GB ECC Unbuffered DDR3 1333 RAM (8Gigs total)
- Norco 4220 (already have)
- 4x 1.5TB barracuda 7200 (already have)
- 4x 2TB Samsung f4's

Now it comes down to the controller:
I believe I have two options

1) 2x USAS-L8i and use the remaining motherboard sata which would connect all 20HDD in the case

2) 1x USAS-L8i and the HP SAS Expander.

Option two is more money but gives me the flexability to add more HDD's if necessary. Also It allows me to connect all drives to the expander and not use the MB satas (not sure if it's a positive but seems simple to me)

Let me know what you guys think. Appreciate the help thus far!!
 
If you have the option of two USAS-L8i i would highly recommend that option over expander; more bandwidth, less potential issues, cheaper, faster, better?

The onboard (chipset-powered) SATA ports are the highest quality (lowest latency) you can have; so they beat anything really. Though most times the southbridge is limited to 2GB/s in bandwidth; 6 * 300 = 1,8GB/s so that's not that bad. The chipset SATA ports are closest to your CPU and have the lowest latency (0.1ms lower than my USAS-L8i in my testings).

8GB RAM is good, much better than 4GB since with only 4 you would have prefetching disabled by default on ZFS. Prefetching works well with 8GB RAM i think; with only 4GB it can actually hurt your speeds.

SuperMicro also has a board with both 6 SATA and 8x SAS ports; and also 2x Intel gigabit NIC. This may be worth a look as it saves you one controller. I believe this one is called Supermicro X8STE-O.

@doublejj0305: even if it has LSI 1068E chip, you need to be able to flash it with SuperMicro IT firmware; else it would function as RAID controller which is useless to ZFS as the controller will deny ZFS access to the disk when it misbehaves. You want a normal HBA or SATA controller; capable of running without RAID but as normal AHCI HBA. If those controllers can do that; then it should work just as well as the SuperMicro USAS-L8i. Note that LSI SAS2008 (6Gbps) chip does not work on FreeBSD yet.
 
Thanks for the reply!
Just put the order in so here is the final build:

- Intel Core i3-530 CPU
- Supermicro MBD-X8SIL-F-O Intel 3420mATX Motherboard
- 2x Kingston 2 x 2GB ECC Unbuffered DDR3 1333 RAM (8Gigs total)
- CORSAIR HX Series CMPSU-650HX 650W 52A on 12 rail
- Norco 4220
- 4x 1.5TB barracuda 7200
- 4x 2TB Samsung f4's
- 2x USAS-L8i
- 1x DVD Drive
- 4xSFF8087 toSFF-8087
- 1xSATA to SFF-8087

Couple of questions:

1) Is it possible to run FreeBSD in a virtualbox? I would like to use the computer for torrents since it will be on 24/7

2)I tried to do some research on the Samsung F4's but couldn't find anything about them being a "fake 4k" drive. even if this is the case what are the potential problems using them on a Raidz. I wasn't clear on what you were trying to point out on the in your first reply. I'm coming from a WHS and never setting up anything remotely close to this. In other words I'm very ignorant on this subject.

3) I saw your new web interface you are putting together (very nice BTW) If I do actually get FreeBSD installed and running (fingers crossed) is it going to be easy to make the switch to the Web interface when it is finalized?

4) When installing FreeBSD, should I install it on an HDD that will not be part of the raidz? I have an spare 60GB SSD laying around I could use. although from what I've read it's not to important when all the server is used for is to serve up movies/music to HTPC's around the house?

I do appreciate your time you have spent on this thread and forum in general. I'm definitely still a "young pond skipper" when it comes to this so I'm sure I will have many more question

J.J.
 
1) Install Transmission from ports in FreeBSD. No need to run another OS just for torrenting. You can install the daemon and use either the web interface from another machine or one of the Windows GUI desktop front-ends that communicate with the server.

2) The only thing I'd watch out for is WD Green drives. I've had nothing but issues with it grinding away spinning up and down while parking the heads endlessly on OS X and FreeBSD.

4) Absolutely. A SSD would make a great boot/os drive. You could also go for another drive and use the SSD as a cache for the ZFS array(s) but this wouldn't be all that useful for serving media such as HD video.
 
Oh feel free to ask; i love explaining these things!


1) Yes i run FreeBSD in Virtualbox and it runs pretty well. If you want FreeBSD to do the ZFS part then you would have to setup your disks as raw disks with the "VBoxManage internalcommands createrawvmdk -rawdisk ...." command. Not sure if this works on Windows hosts, but i would not recommend running on a Windows host, likely because all the cool CPU scheduling and other stuff on FreeBSD wouldn't work very well.

I'm not very familiar with virtualization, but having it as a dedicated box would weed out any reliability issues, performance issues and dependency on a specific product. On the other side, a VM is very flexible and easy to setup and cheap as it requires less hardware to virtualize two OS on one system, or even more.



2) Like the WD EARS, the Samsung F4 series have 4096 byte physical sectors but emulate 512-byte sectors; thus they lie about their sector size so to remain compatible with 10-year-old Windows XP. And i haven't heard of a jumper allowing to set the correct sector size. The jumper on WD EARS is used to shift LBA by making 64 sectors per track instead of 63 sectors per track.

So what are the issues with 4K disks when used with ZFS? Two issues:
- start alignment
- end alignment

Start alignment usually is always good when you have an aligned partition on the disk, or when not using any partitions at all and feeding ZFS the raw device without any partitions. That last option might not be a good combination with Windows; as Windows would prompt you to 'initialize' these drives without partitions and if you do, it will corrupt them by overwriting with a fresh MBR partition table. This can happen i think if your host is Windows and you use Virtualbox FreeBSD+ZFS. So be careful! And one more reason to avoid windows as host-OS.

End alignment is more complex. With 4K-disks using 512-byte sector emulation, we want to avoid having the drive ever to use its emulation. Let's show an example to make this clear:

Example 4K drive layout:
|01|02|03|04|05|06|07|08|09|10|11.....

Each number is a sector as the drive tells the OS; 512-bytes. But the drive has physical sectors of 4096 bytes. Now look at the example and imagine what would happen if we read sector 1 through 4, or 5 through 8. We would read exactly 4 sectors at once; 4 x 512 bytes = 4096 bytes; one physical sector. So as long as we read in chunks of 4 sectors at once; we're good!

Now imagine what would happen if we read sectors 3 through 6; also 4 sectors long but they overlap two physical sectors. The physical sectors start at 1 to 4, and 5 to 8, etc. If you read only a part of such a physical sector, then the whole 4096 byte physical sector will have to be read; even if only one chunk of 512 bytes is requested.

So what? Reading 4K instead of 0.5K; is that such a big deal? No, it's not. When reading the 'wrong' way; the drive won't have much performance issues at all even when it has to emulate its 512 byte sectors and read more than it actually has to.

But the problems start when WRITING. When you write to only part of a physical 4K sector, the drive has to emulate just like reading. But it cannot just write the whole 4K sector because it doesn't know the contents of the data you are not writing. Complicated? Check the example graph again. If you write to say only sector 2, then the drive has to write sectors 1 through 4. But it doesn't know the contents of sector 1, 3 and 4 yet; it has to READ those first; then write the whole 4K sector. Just to write 512 bytes so much trouble!

So writing in anything but chunks of 4 kilobytes (i.e. 128KiB would be good as well) will cause heavy performance degradation. Now back to ZFS: how does this affect this issue?

ZFS RAID-Z uses variable stripesize; in essence it spreads 128KiB or even less over your disks. Assume you have a 4-disk RAID-Z with one parity 'disk'; now one 128KiB write comes, gets split due to RAID-Z to 128 / (4 - 1) = ~34KiB.

This 34KiB is not a multiple of 4K; it would have been aligned if it was 32KiB or 36KiB; but 34 is bad! Now for every write ZFS does, the 4K-sector-with-512b-sector-emulation will emulate the last 2 kilobyte write, having to read and thus slowing down considerably.

Solutions?
- use geom_nop to make the drive report as having 4096B / 4K sectors; all problems disappear; until you reboot since geom_nop is not 'remembered' across reboots/shutdowns. My Mesa project might have an answer for this last issue.
- tune the recordsize so the disks get only aligned chunks of multiples of 4K when writing. Note that due to RAID-Z works you cannot rule emulation out if you do smaller writes than 128KiB. But at least it wouldn't always be slow; now it can just write like normal 512-byte sector drives do.
- use ZFS mirroring instead of RAID-Z; problems disappear as long as your start partition is aligned or using raw disk.



3) Yes you can switch between FreeBSD/OpenSolaris and my Mesa project. Things to keep in mind, though:
- You cannot migrate a ZFS pool of higher version (i.e. 15) to a lower-version ZFS running OS like FreeBSD 8.1/Mesa which runs ZFS pool version 14.
- Take care on how you prepare your disks. Using GPT partitions with a GPT label, or using GEOM_LABEL, is recommended but not strictly required.
- My project is not yet in a usable state; i'm releasing 0.1.6 in about a week but still at least a month away from 0.2.0; the first semi-stable release.

Handling your disks correctly would be the thing you can prepare for if you want the option to switch to my project when it becomes usable/stable/desirable by you. If you would opt for a real FreeBSD install then i can give you all the guidance you need for setting up your disks with gpart; which creates GPT partitions with labels. The labels are important to identify your disks; and my project basically built around these labels as they are incredibly useful i think.



4) Upcoming 0.1.6 version of my Mesa project will have ability to install directly to RAID-Z or any ZFS array. This removes the need for a separate system disk. There are still issues: how do you recover/investigate if your array is so damaged that it will not boot? Then you can use the USB or LiveCD to check the status of the array; circumventing your ZFS-on-root install.

The advantage here is that you don't need another disk just for the OS. It only takes about 220MB of your ZFS array. Any member disk is bootable and boot to the same OS. In future you would also be able to boot off RAID-Z3 or triple parity RAID. There's already a ZFS v28 patchset which is experimental but will be the basis of ZFS v28 support in FreeBSD 9. There is a virtualbox image available for easy testing. Ask me if you want the URL.

The SSD is still very useful though! You can use it for two purposes (or both):
- L2ARC cache device: this will speed up your array when reading random parts; i.e. if you store firefox profile on your ZFS array this would speed up the random reads going to your ZFS array. A small SSD could make the entire array feel as fast as an SSD. Highly recommended! But can be write-heavy on your SSD.
- SLOG ZIL log device: use as the 'log device' so the HDDs themselves are not used for this function. This speeds up writes and can significantly benefit write performance! Warning: if you lose your SLOG device the entire array becomes inaccessible with ZFS versions prior to 19. Normally, the SLOG would also have the redundancy of the array (mirroring or RAID-Z). There are also issues with current SSDs which can not write safely in the event of a sudden power interruption; then they can cause corruption and potentially cause alot of harm to your array. I would wander this route when you get access to ZFS pool version 19 or later.

The SSDs would benefit your ZFS array for small random accesses; which HDDs suck at but by combining SSDs and HDDs intelligently we can create the best of both worlds: a small SSD can accelerate a multiterabyte array by a good margin. But if all you do is write and read large files without ever modifying parts of those files (causing fragmentation); then this may not be of particular interest to you. You can use the SSD for something else or as normal ZFS storage.

If you don't understand a portion of text then simply quote it and i'll explain it better. :)
Some things are rather complex, but still logical if you can set your mind to it. If not, then just look for the conclusions and go on those.

Hope you have a good time with ZFS!
Cheers!
 
Last edited:
Wow that is one hell of a reply and appreciate every word!

1) I'll just use it as a stand alone, you don't sound too confident in running a VB thru windows. This is really no big deal. My top priority is my data.

2) I'm starting to piece this together and came up with my own SOLUTION: Cancel my order for the F4's and get another another brand of 2TB "green drives" that play nicely with ZFS. Any suggestion on 2TB drives? All drives manufacturer drives fail so I could care less which brand, the only thing they have to be is 2TB and "low wattage"

4) Thats great so your saying I can just run the ZFS off a thumb drive using your project and I don't have to learn FreeBSD line commands!!!! I know your project is still in pre-stages but once the hardware arrives I'm going to install Mesa project and skip the FreeBSD install all together. I have a copy 1:1 copy of all the Data I'll be putting on there so if something fails it's no big deal. I'll hold on to the SSD and maybe use it for L2ARC for the small amount of pictures and music I have.

I can't THANK you enough for all your input and work on your project. Once I start running MESA I'll definitely help in any way I can (which might not be much ;) )
 
About the Samsung F4 disks: it's not that i don't recommend them. They are very low power since they have only 3 platters; and due to 666GB-per-platter the data density is very high translating to high sequential speeds. This is the fastest 5400rpm drive out yet!

If you would buy five of these instead of four, you should have good write speeds with RAID-Z; in theory though, i haven't verified this myself. What would happen? With 5 disks in RAID-Z, you have 4 effective data disks. Divide 128 by four and you get 32KiB per disk; that's aligned! This should mean that you get out-of-the-box good speeds with 5-disk RAID-Z on 4K sector disks; but not on 4-disk RAID-Zs as you cannot divide 128 by three and get an aligned number.

So this would still be an option; but a risky one. If you just want a proven solution, scout for 4-platter 500GB 5400rpm green drives like WD EADS and Samsung F2EG. Not as fast, slightly more power usage, but no potential issues. It's up to you. :)
 
Oh haven't seen the coupon discount. Yes that would be a large difference. I have no experience with this drive or heard of users using it with ZFS; but no reason to suspect anything wrong on them. They seem like a really cheap 2TB option and without the 4K sector performance penalties in some setups.
 
A whopping 39% of 289 newegg reviewers rated the Seagate Barracuda LP ST32000542AS one or two eggs. That is enough for me to not touch the thing.
 
Hmm yes doesn't look too good; and quite a few people entered a vote making this a fair indication. But it does have 3 years warranty.

As long as you do backups and redundancy properly you should have no problems with disk failure; as long as you can replace them within a fair amount of time. I know some HDD manufacturers allow something like paying ahead with your creditcard for a new disk, send your failed RMA disk and if it's really verified failed then the amount gets refunded on your card. That would let you able to get a new drive fairly quick. And a backup should always protect your data in case you break the redundancy of your main array. Breaking the redundancy on both of them within a few days is highly improbable.

But you're right that these wouldn't be my new disks. But price is another thing, and if they just work and ZFS+backups would protect his data then i can't say i blame him if he prefers the lower price, relying on the 3 years warranty to get replacement disks.
 
Also, the Seagate specs <1 in 10^14 BER. The WD Green and the Samsung spec the <1 in 10^15. The 2TB Samsung F3EG has the best newegg reviews of all the 2TB drives with sufficient number of reviews, and I would guess the F4 will also collect similar good reviews (most Samsung HDDs on newegg have good reviews). But I've never tried Samsung's return procedure. I know WD has a very easy RMA procedure (your choice of advance return or normal return). So I'd go with the Samsung or WD, depending on whether I weighted good reviews or good return service more highly.
 
You guys are making me more indecisive than my wife. I'm going to pony up the cash and get the EADS. I've dealt with Western Digital on RMA's and they are great.

So the new array will be raid-z2 with 8x WD 2TB Green drives, the 1.5 seagates I have currently will be used for storage in a safe.
 
Well i can believe differences between brands or even models can be huge. But to measure this only in units of a factor of 10 i think this is a very rough measurement.

Unlike other specs which can be verified like power consumption in various states, the BER is hard to quantify without doing an investigation involving many drives. The information Google released would be interesting, as they mentioned severe differences in failure rate between HDD vendors, without naming them though.

Anyway, my point: with a spec that is quite unverifiable it may become subject to interpretation or creativity of the marketing department. You can see that happening with WD moving TLER to RAID edition disks only by firmware. So i'm not sure how helpful these BER specs are.
 
You must log in or register to reply here.
Back
Top