My first ZFS-setup

J

jocke

n00b
Joined
Oct 4, 2010
Messages
32
Hello [H],

I bet there's 2134512 threads about this, but figured I'd want my own thread.

Until now, I've used Windows XP, with two PCI raid controllers, and a few external USB-disks, ending up with these hard drives;

1x WDC WD2500AAKS-00F0A0 (250GB)
1x WDC WD3200KS-00PFB0 (320GB)
2x WDC WD10EADS-00L5B1 (1TB)
6x WDC WD10EACS-00ZJB0 (1TB)
3x WD 10EAVS (1TB)
1x WD 3200JB (320GB)

Which makes up 15 partitions (it's a nightmare -- I know).

I've wanted to move away from this setup for quite some time, but I've just postponed it again and again. Now I'm hoping that I can do something about it during the Christmas holidays.

I've heard so much good about ZFS, so this is the direction I'm heading. I've bought the HP SAS Expander, LSI SAS 9211-8i, and the cables.

The disks that I currently have, are all full, so I have to buy a few new ones, and move stuff over step-by-step (move, erase old hard-drive, add to pool, move, and so on). I'm not planning to use the external disks anymore, or the 250GB/320GB ones, so I'll need to buy at least 5TB worth of new hard-drives (3x 1TB + 2x 320GB + 1x 250GB + 1TB extra).

So, the question is this; how should I proceed? I know theres lots of lots of threads regarding 4kB sector size versus 512b. Since this is my first setup, I'm not sure if I wan't to start tweaking all kinds of things to get 4kB-disks working neatly, or? Should I just buy disks with 512b to be on the safe side? How will this work if a disk fails in 2 years, and 512b disks isn't available anymore, so that I have to add one with 4kB whilst the other disks in the pool are still 512b?

Currently I'm looking at WD2001FASS, but they have a rather steep price to be "only" 2TB disks. Maybe it's enough with 5400rpm disks? Any suggestions?

I guess I also should buy myself some more memory? The box currently runs a C2D E7400 with 2GB DDR2 memory. The plan is to use FreeBSD -- normal version, or the one sub.mesa has made?

Anything else I haven't thought of? Anything else I should be aware of?
 
You should not be buying old 250GB disks anymore; doing so would be a waste.

Instead, consider using the 250GB and 320GB as external backups rather than primary storage; they use alot of power per gigabyte of storage; whereas 2TB green disks have a very good ratio of power per gigabyte of storage.

So buying 2TB disks ans using your existing 1TB disks in either one or two pools would be the most common setup. You can have:

vdev1: 10x 1TB in RAID-Z2 (these you already have)
vdev2: 6x 2TB in RAID-Z2 (these you can buy now or later)

So you can mix the capacities of your pool, but i would not bother with the 320GB/250GB disks and use those for another purpose instead. These old disks are quite slow and use alot of power as well.

You may have noted that i said 10x 1TB and now 11 while you have indeed 11 drives. Well 10 disks in a RAID-Z2 works better especially for 4KiB sector disks (yours are 512B tho) but also you could leave one as hot spare. With these kind of disks you may want this feature, especially since the EAVS has basically TLER=0sec; it would develop bad sectors in a more rapid rate than normal EADS/EARS disks. The whole idea of ZFS is that you do not need TLER-enabled disks, including "A/V" disks which is what that EAVS is. You can still use them, however, but i would recommend RAID-Z2 over just RAID-Z as i consider these disks suspect and need additional redundancy to prevent issues with Bit-Error-Rate (BER) eating your data.

Samsung F4 2TB 5400rpm is the preferred ZFS disk at the moment; even though its 4KiB sector it achieves very good performance at very low power and low cost. Be sure to flash with latest firmware to fix a corruption issue with this drive, though.

Your CPU is okay, but 2GB RAM is too little for optimal performance. Buying DDR2 is expensive though; consider the marginal extra cost of buying a new platform; since DDR3 will be cheaper. Buying 2 sticks of 4GiB DDR3 ECC memory would be quite cheap and you get both ECC memory back and you get 8GiB memory for the same price as 4GiB DDR2 non-ECC memory.

Going DDR3 does mean a new motherboard and CPU, though. Personally i would wait until Ontario/Brazos platform, and get something like this:

gbt_fusion_zacate_mobo.jpg


But buying one more stick of 2GB DDR2 memory to 4GB total would be a more easy alternative. Note that ZFS, at least on FreeBSD, achieves optimal performance starting with 6 or 8 GiB of RAM. With 4GiB and less you need to perform tuning to get both acceptable performance and stable operation.

As for running my project or vanilla FreeBSD; i suggest you try my project first to get familiar with some stuff; but do explore FreeBSD and get comfortable with the command line if you want to get more under the hood; you can always install vanilla FreeBSD and import your pool created with ZFSguru in vanilla FreeBSD instead. The advantage of my project; besides a GUI, would be easy installation on ZFS ("root-on-ZFS") and easy system upgrades to other FreeBSD versions. Whatever you choose, you have all the flexibility and configurability you want.
 
Hi sub.mesa, thanks for your reply.

sub.mesa said:
You should not be buying old 250GB disks anymore; doing so would be a waste.

Instead, consider using the 250GB and 320GB as external backups rather than primary storage; they use alot of power per gigabyte of storage; whereas 2TB green disks have a very good ratio of power per gigabyte of storage.
Click to expand...
Yes, the 250GB and 320GB disks I'll throw away (I don't need them anymore). They've been around for ages, so a replace is in due time. To summarize, in my new setup, I'll only be using the following of my old hard-drives;

2x WDC WD10EADS-00L5B1 (1TB)
6x WDC WD10EACS-00ZJB0 (1TB)

You may have noted that i said 10x 1TB and now 11 while you have indeed 11 drives. Well 10 disks in a RAID-Z2 works better especially for 4KiB sector disks (yours are 512B tho) but also you could leave one as hot spare. With these kind of disks you may want this feature, especially since the EAVS has basically TLER=0sec; it would develop bad sectors in a more rapid rate than normal EADS/EARS disks. The whole idea of ZFS is that you do not need TLER-enabled disks, including "A/V" disks which is what that EAVS is. You can still use them, however, but i would recommend RAID-Z2 over just RAID-Z as i consider these disks suspect and need additional redundancy to prevent issues with Bit-Error-Rate (BER) eating your data.
Click to expand...
All the three 1TB EAVS-disks are external, and I'm not gonna use them in the new setup (I'll use them for backup of my laptop and so on). That leaves me with 8x 1TB disks. What would the best usage for those be? 8 in a RAID-Z2? Is there some kind of "formula" to calculate the best combination of drives? I mean; what is the general rule?

Samsung F4 2TB 5400rpm is the preferred ZFS disk at the moment; even though its 4KiB sector it achieves very good performance at very low power and low cost. Be sure to flash with latest firmware to fix a corruption issue with this drive, though.
Click to expand...
So, I guess I'll just get 6 of those, then? (if that's the recommended size of a RAID-Z2 vdev?)

Your CPU is okay, but 2GB RAM is too little for optimal performance. Buying DDR2 is expensive though; consider the marginal extra cost of buying a new platform; since DDR3 will be cheaper. Buying 2 sticks of 4GiB DDR3 ECC memory would be quite cheap and you get both ECC memory back and you get 8GiB memory for the same price as 4GiB DDR2 non-ECC memory.
Click to expand...
Ah. Hahaha, lovely. Do you have any quick suggestions as to what works well with FreeBSD? (that is, motherboard, CPU and memory?). I don't need any fancy, expensive stuff -- just something that can run the ZFS-setup decently.

As for running my project or vanilla FreeBSD; i suggest you try my project first to get familiar with some stuff; but do explore FreeBSD and get comfortable with the command line if you want to get more under the hood; you can always install vanilla FreeBSD and import your pool created with ZFSguru in vanilla FreeBSD instead. The advantage of my project; besides a GUI, would be easy installation on ZFS ("root-on-ZFS") and easy system upgrades to other FreeBSD versions. Whatever you choose, you have all the flexibility and configurability you want.
Click to expand...
I have no intentions to install the system on ZFS -- I'll get a seperate disk for the OS. I guess I'll try out your version, I just need to get the hardware-part sorted out first. (-:
 
Okay. I've now ordered 2GB more DDR2-memory (it cost me $50, so no big deal). I'll see if that is sufficient, and if not, I'll order a new motherboard, CPU and some DDR3-memory.

I also ordered 7 of those Samsung-drives you mentioned. 6 to the vdev, and the last one for the OS.

What happens if I have to replace one of the 512b-disks? If it takes a year before it fails, there might not be 512b disks available anymore, and mixing 512b and 4K disks in the same vdev doesn't work, right? (or at least degrades the performance a lot?).
 
8x 1TB in RAID-Z2 will work fine.

For 512-byte sector disks:
RAID-Z: between 3 and 8 (more would not be safe)
RAID-Z: between 6 and 12

For 4K sector disks:
RAID-Z: 5-disks (9-disks also fast/good option but not that much redundancy)
RAID-Z2: 6-disks or 10 disks.

FreeBSD should work on most hardware; you should use a controller that's supported but yours should work well. Make sure it is not running in RAID "IR" firmware mode, though.

As for the OS; you can just use a USB pendrive; no need to have a 2TB disk store 250MB for the OS only. This also saves you power. If you go this route, you can use the 7th 2TB disk you bought as separate pool as backup; these disks are ideal backup disks! The EAVS you use as backup may not be ideal for this task, because these disks would develop bad/unrecoverable sectors more rapidly than other "normal" disks and especially 4K disks; 4K disks have better protection against bad sectors because they have larger ECC (error correction) data for each sector. The EAVS drives are poor for backup tasks because they give up IMMEDIATELY whenever they can't read a sector; they don't bother trying to recover they forfeit their ability right away; such is the design of these "A/V" drives which may never stutter or have high latencies due to recovery; so these drives would fail the sector over trying to recover them, making them 'hickup-free' for audiostreaming tasks, where a bad sector means you delete that file and re-create it again; the data on these disks should be temporary and replaceable. That said, if these are backups then you already have a main array too, but still you would want to know this i think.

Replacing a 512-byte HDD with 4K-sector-emulation HDD would work; since these disks emulate 512-byte sectors instead. If you use the sectorsize override feature, you would have better performance if you ever need to replace a 512-byte sector HDD by 4K HDD.
 
Stay away from WD and get 6x Samsung F4EGs. Patch with firmare fix and you are on your way! My ZFS system is now WD-free and it's a freaking big difference.
 
jocke said:
All the three 1TB EAVS-disks are external, and I'm not gonna use them in the new setup (I'll use them for backup of my laptop and so on).
Click to expand...

The WD10EAVS drives should be fine. That is a discontinued model, but you can still find the spec sheet on Western Digital's web page:

http://www.wdc.com/wdsearch/?sc=&sl=en-US&sq=WD10EAVS
http://www.wdc.com/en/library/sata/2879-701229.pdf?wdc_lang=en

You can see that the WD10EADS and WD10EAVS have identical specifications. In particular, "Non-recoverable read errors per bits read" is <1 in 10^15 for the WD10EAVS, That is 125 TBytes, on average, per non-recoverable read error. The chances are better than 50% that you could read the entire drive 62 times before you see an unrecoverable read error.

I've seen people posting speculations about some models of WD HDDs and how they behave, but I have never seen anyone with any evidence to dispute what Western Digital specifies for the drives. I'll take Western Digital's specifications over idle speculation.
 
You don't exactly have to search long to find people with WD GP problems. The problem isn't that they loose data - they have a tendency to go SLOOOOW and halt the whole array. Put 4k/512 emulated drives into the mix and the problem is worse...
 
Oh, and try to get Solaris running on the ting before you try FreeBSD if you value perfomance... :)
 
Flintstone said:
You don't exactly have to search long to find people with WD GP problems. The problem isn't that they loose data - they have a tendency to go SLOOOOW and halt the whole array.
Click to expand...

What array? He said they were external and he was planning to use them for backup. As I said, they should be fine.
 
Thanks for all the replies.

sub.mesa said:
As for the OS; you can just use a USB pendrive; no need to have a 2TB disk store 250MB for the OS only. This also saves you power.
Click to expand...
All valid points, but I prefer having permanent storage inside the machine (which includes the OS-disk). I don't need 2TB for this, but the local computer-shop didn't have anything lower for a OK price. I'm thinking that I could partition it, or use it as temp-download space. It's not going into the ZFS-setup, though.

The EAVS you use as backup may not be ideal for this task, because these disks would develop bad/unrecoverable sectors more rapidly than other "normal" disks
Click to expand...
Does it matter if they're in a ZFS-setup? I mean; the worst that could happen, is that I'd have to replace the disk?

Replacing a 512-byte HDD with 4K-sector-emulation HDD would work; since these disks emulate 512-byte sectors instead.
Click to expand...
Isn't this what the WD Advanced Format-disks does? I mean; I thought this was "bad"?

If you use the sectorsize override feature, you would have better performance if you ever need to replace a 512-byte sector HDD by 4K HDD.
Click to expand...
Could you elaborate on this? (-:


Flintstone said:
Stay away from WD and get 6x Samsung F4EGs. Patch with firmare fix and you are on your way! My ZFS system is now WD-free and it's a freaking big difference.
Click to expand...
I've never, _ever_ had a WD-disk crash on me, and I've been using them for longer than I can remember. So it's not like I'm doing this ZFS-thing because I have problems with my current setup, but merely because I'd like to at least have _some_ redundancy in case something should happen. Getting rid of Windows is also a really nice side-effect!


john4200 said:
The WD10EAVS drives should be fine.

[...]

I've seen people posting speculations about some models of WD HDDs and how they behave, but I have never seen anyone with any evidence to dispute what Western Digital specifies for the drives. I'll take Western Digital's specifications over idle speculation.
Click to expand...
Yes, I've never had any problems with WD-disks before. Ever. Maybe I'll consider using 10x1TB after all.


Flintstone said:
You don't exactly have to search long to find people with WD GP problems. The problem isn't that they loose data - they have a tendency to go SLOOOOW and halt the whole array. Put 4k/512 emulated drives into the mix and the problem is worse...
Click to expand...
Well, yes, I've figured this part out already, but it's not like I'm gonna replace those 11x 1TB-disks without giving it a shot first.


Flintstone said:
Oh, and try to get Solaris running on the ting before you try FreeBSD if you value perfomance... :)
Click to expand...
Well, if I can reach 1Gbps (125MB/s) of speed, that'll be enough. I don't know what I should expect from a setup like this, but I figure 125MB/s should go fairly well. (Or?).


john4200 said:
What array? He said they were external and he was planning to use them for backup. As I said, they should be fine.
Click to expand...
Well, yes, the plan was to use them for backup (or whatever), but that was based on the feedback I got in here (and partly because they are external, and I'd have to open them up to put them inside the machine if I decide to use them), but if they are OK after all (I mean, it's ZFS -- I shouldn't have to worry about loosing a hard-drive?), I might consider using them.


I have a practical question, btw. Since the old disks are full, the plan is to setup the system with the 6x2TBs in a vdev (RAID-Z2) first. Then I'll copy over the contents of a few of the old drives. I could then start building a new vdev (RAID-Z2) with only a few of the old drives, and expand it as-I-go? Or are there limitations that I need to keep in mind here?
 
jocke said:
Since the old disks are full, the plan is to setup the system with the 6x2TBs in a vdev (RAID-Z2) first. Then I'll copy over the contents of a few of the old drives. I could then start building a new vdev (RAID-Z2) with only a few of the old drives, and expand it as-I-go? Or are there limitations that I need to keep in mind here?
Click to expand...

Expand "it". If by it, you mean expand the vdev, then no, you cannot. You can expand a pool by adding vdevs. So, you could add another 6-drive raidz2 vdev. But you cannot expand a 4-drive raidz2 vdev to a 5- or 6-drive raidz2 vdev. This is a ZFS limitation.

As for the WD10EAVS in ZFS, I'd just give them a try. There is nothing in the specifications that indicate that you should not do it. There could be compatibility problems (or not), but that is almost always the case, unless you build everything from a hardware compatibility list that someone has extensively tested already.
 
john4200 said:
Expand "it". If by it, you mean expand the vdev, then no, you cannot. You can expand a pool by adding vdevs. So, you could add another 6-drive raidz2 vdev. But you cannot expand a 4-drive raidz2 vdev to a 5- or 6-drive raidz2 vdev. This is a ZFS limitation.
Click to expand...
Yes, by "it" I meant the vdev. Hmm, since I need 10TB of free storage to move everything, I have an excuse to buy more than the 6x 2TB's. :D
 
jocke said:
Yes, by "it" I meant the vdev. Hmm, since I need 10TB of free storage to move everything, I have an excuse to buy more than the 6x 2TB's. :D
Click to expand...

With 2TB disks, I'd go for a raidz2 instead of raidz1. 2TB disks can take a looong time to rebuild. I have 2 x 8 disk raidz2's in one pool, using 2TB disks. Resilvering takes roughly 8hrs (too long), hence the move to raidz2.

Better make it 7 disks :D
 
I don't know how the used marked for drives are in your area, but here it's not really a problem to sell a 1TB drive for half the price of a 2TB drive. As crazy as it sounds, this has been my way to go from 7x250GB + 7x320GB -> 14x500GB -> 14x 750/1000 GB -> 14x1000/1500GB -> 17x 2TB over the years...

Just keep it in mind - it might reduce the number of drives you need. I try to not keep drives over 1 year to avoid problems.

Sorry about the WD bashing - they're not that bad - just the GP line I've had problems with (not lost data - just drives that grinds to a halt, TLER problems and IntelliPark issues).
 
tormentum said:
With 2TB disks, I'd go for a raidz2 instead of raidz1. 2TB disks can take a looong time to rebuild. I have 2 x 8 disk raidz2's in one pool, using 2TB disks. Resilvering takes roughly 8hrs (too long), hence the move to raidz2.

Better make it 7 disks :D
Click to expand...

I second this. RAIDZ2 is best - my 10x 2TB setup uses 11 hrs to scrub (98% or so full). But if your data isn't life changing critical you can get a long way with normal RAIDZ. I know sub.mesa recommends using either 6 or 10 4k drives in a RAIDZ2 configuration for optimal speed, but speed might not be a huge concern for you. I haven't measured the performance hit myself, but others have.
 
Flintstone said:
tormentum said:
With 2TB disks, I'd go for a raidz2 instead of raidz1. 2TB disks can take a looong time to rebuild. I have 2 x 8 disk raidz2's in one pool, using 2TB disks. Resilvering takes roughly 8hrs (too long), hence the move to raidz2.
Click to expand...
I second this. RAIDZ2 is best - my 10x 2TB setup uses 11 hrs to scrub (98% or so full). But if your data isn't life changing critical you can get a long way with normal RAIDZ. I know sub.mesa recommends using either 6 or 10 4k drives in a RAIDZ2 configuration for optimal speed, but speed might not be a huge concern for you. I haven't measured the performance hit myself, but others have.
Click to expand...
RAID-Z2 is what I had in mind, yes. As mentioned, I've bought 7x 2TB (which I'm picking up tomorrow, btw). 6x 2TB in RAID-Z2, and the 7th as OS-disk. If I've understood it correctly, I'll end up with 4x 2TB of usable drives. Since I currently have ~9TB of actual data on the 11x 1TB drives, I don't have enough space on the new setup. Since I can't expand the vdev, I'll either have to use two smaller pools with the 1TBs, or buy enough disks to build another vdev. I'm thinking the latter, but I'll have to think it through.
 
Flintstone said:
I don't know how the used marked for drives are in your area, but here it's not really a problem to sell a 1TB drive for half the price of a 2TB drive. As crazy as it sounds, this has been my way to go from 7x250GB + 7x320GB -> 14x500GB -> 14x 750/1000 GB -> 14x1000/1500GB -> 17x 2TB over the years...
Click to expand...
Well, I don't think theres much marked for that in Norway. You see, we have this "consumer purchase law" here, which gives us the right to claim repairing of broken products (f.ex. like a hard-drive) for 5 years after it's purchased. If they can't fix it, you're entitled for a replacement. This is regardless of whatever guarantees the producer of the product has. With this law, there's really no need to worry about broken disks; I'll just get new ones (and if the old ones aren't available anymore, you'll get an equivalent one). Kinda neat. :D
 
jocke said:
Well, I don't think theres much marked for that in Norway. You see, we have this "consumer purchase law" here, which gives us the right to claim repairing of broken products (f.ex. like a hard-drive) for 5 years after it's purchased. If they can't fix it, you're entitled for a replacement. This is regardless of whatever guarantees the producer of the product has. With this law, there's really no need to worry about broken disks; I'll just get new ones (and if the old ones aren't available anymore, you'll get an equivalent one). Kinda neat. :D
Click to expand...

But how do the prices compare to prices in the US? (i.e., using the current exchange rate). I'd bet the manufacturers charge higher prices in Norway than they do in the US, in order to pay for all the "free" replacements.
 
john4200 said:
But how do the prices compare to prices in the US? (i.e., using the current exchange rate). I'd bet the manufacturers charge higher prices in Norway than they do in the US, in order to pay for all the "free" replacements.
Click to expand...
694 NOK including VAT for one those Samsung F4EGs. That's 116 USD using Google ("694 NOK IN USD").
 
jocke said:
Well, I don't think theres much marked for that in Norway. You see, we have this "consumer purchase law" here, which gives us the right to claim repairing of broken products (f.ex. like a hard-drive) for 5 years after it's purchased. If they can't fix it, you're entitled for a replacement. This is regardless of whatever guarantees the producer of the product has. With this law, there's really no need to worry about broken disks; I'll just get new ones (and if the old ones aren't available anymore, you'll get an equivalent one). Kinda neat. :D
Click to expand...

Hehehe... This was funny :) I'm Norwegian :)
 
jocke said:
694 NOK including VAT for one those Samsung F4EGs. That's 116 USD using Google ("694 NOK IN USD").
Click to expand...

I just (this week) sold 7 crappy WD GP drives (1.5TB) for 450 NOK each and 5 Samsung F1 1TB drives for about 350 NOK each. And bought myself a few (7) more 2TB drives at 700 NOK (retail)...

Now my server is maxed out at 17 2TB drives (+ 2.5" system drive) so I'll probably go atleast a year or so before I upgrade my system again... :)
 
Okay. I've now assembled it all. It took some more time than anticipated, due some major recabling in the tower, and the need for a second PSU (which had to be cabled to power on with the main PSU).

It's all up and running, and until I've tested stuff properly, I've unplugged all the old hard-drives, just to make sure I won't f*ck them up :-P

First issue; I wanted to upgrade the firmware on my 9211-8i. However, it wouldn't let me flash it with the P8 IT-firmware (it said; "ERROR! Cannot Flash IT Firmware over IR Firmware!"), so I just flashed it with the P8 IR-version. Is this a big issue? From what I've read, it passes the disks just fine to the OS even with IR-firmware (if I'm to believe the comments on this page: http://www.natecarlson.com/2010/08/23/lsi-command-line-utility-for-sas2-non-raid-controllers/ ).

I haven't actually made any tests regarding ZFS yet -- that's up next.
 
Woah!

After upgrading to P8, the LSI-controller now states that I have 8 enclosures (whereas it only stated 1 on the old version), totaling 8x6 disks (even though I only have 6 disks connected).

The sas2flash-utility won't let me downgrade either -- any suggestions?
 
Hm i didn't flash my own SuperMicro with any firmware; it came with IT-firmware. Perhaps you can use the LSI IT firmware or supermicro IT firmware and try another flash utility? I've heard commonly that people flash their Intel SASUC8i coming in IR raid mode to IT mode instead. So this should be possible.

The problem in IR mode is that you still may get dropped disks, theoretically at least like other hardware RAID; i never used LSI controller in IR mode. But IT mode would be the most ideal; and probably gives you clear SMART throughput and TRIM capability and allows you to set AAM/APM and other stuff, which may not be possible in IR mode. At least not via the software, i think.
 
Finally.

I've now been able to flash it to P7 IT firmware; I had to boot it into native DOS, since I had to use "sas2flsh.exe" (and not "sas2flash.exe") to be able to erase the flash (to downgrade) and to overwrite the IR firmware with IT. I spent ages getting DOS to find my CD-ROM so that I could access the firmware-files/utilities.

Now the expander shows up as one enclosure. I discussed it with LSI over email, and they could confirm that the HP SAS expander (2.06) seems to be incompatible with the 9211-8i running P8 firmware.

Now I can actually start to use this system :-P
 
Good trouble shooting jocke! These little things can be annoying i'm sure, hope your next experience will be better!
 
A quick question; if I, in the future, need to replace a broken 1TB drive in a vdev, and there are no 1TB drives available (or they are expensive), would I then suffer a large performance-drop by using a larger drive? (like 2TB or 3TB).
 
Last edited:
You must log in or register to reply here.
Back
Top