ZFS fileserver/backup build assistance

I

invade

n00b
Joined
Apr 16, 2012
Messages
8
Hi all,

First post, but I've been a lurker around just never got to making an account on the forums. One thing though, I admire most of what you all have in terms of builds hella nice rigs hopefully I can reach those ranks at some point. Anyhow! I plan on making a fileserver and then I'll have crashplan run overnight to backup.

Parts
Case: Fractal Design Define XL Titanium E-ATX Or. Lian Li PC-A77FB
PSU: Corsair Enthusiast Series TX850M
RAM: Corsair CMZ16GX3M4A1600C9 Vengeance 16GB
CPU: Intel Core i3 550 Dual Core Processor Clarkdale
MOBO: Gigabyte Z68MA-D2H-B3 mATX LGA1155 Z68
Raid: LSI 9265-8i going raidz2
Bay: Supermicro Black 3X5.25 to 5X3.5IN SATA Drive Bay
HD: 4x Western Digital Caviar Black 2TB
OS: FreeBSD

Would these parts be sufficient for a ZFS system? I do plan on adding more drives over time. Please let me know if any of these parts would be an issue or if there's another alternative that I can go for?
 
The CPU is incompatible with that motherboard. You need to use a LGA 1155 CPU, not a LGA 1156. So go for the Core i3 2120.

PSU wise, that Corsair isn't that great. I'd spend the extra cash for a better quality modular PSU:
$145 - NZXT HALE90-750-M 750W Modular PSU
$140 - Seasonic M12II 750 SS-750AM 750W Modular PSU
$154 - Corsair HX750 750W Modular PSU
$160 - Seasonic X750 Gold 750W Modular PSU

If you don't need modular cables, you'll be fine with these PSUs:
$105 - Corsair 750TX V2 750W PSU
$125 - Corsair 850TX V2 850W PSU
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
What's wrong with the Corsair TX850M? It's stable up to it's capacity at 45c, and has decent ripple and noise suppression. Yes, the TX850 v2 would be a superior power supply (Seasonic vs CWT), but for the load purposes even up to 24 disks on an i3, I don't think it's going to matter.

My recommendation would be for an ECC build rather than a desktop build. Though ZFS is end to end CRC, it doesn't validate the input data from memory, so there is still a chance for data loss if the source data has been flipped to begin with. Have you considered AMD? Unbuffered ECC and a cheap AMD build is going to be sufficient for ZFS on FreeBSD or Solaris for a 730MB/s throughput target. You won't even need an SSD ZIL at 16GB of RAM. I'd also consider 5 disks over 4 for raidz2 to offset the parity space loss. With 4 disks, you're getting the same disk space impact as a mirror.

For future expansion, please remember that you only have two upgrade paths for your pool. For a smaller array as planned, it's simple enough to plug in higher capacity drives one at a time and resilver the array, until the final drive is replaced and the size bump happens. But if you really do intend to add more disks as you go, having 6-8 disks per vdev at raidz2 is pretty optimal, because it'll get maximum throughput on a PCI-E HBA, you can replace fewer total disks for a size bump, and you don't run into the problematic situation of a huge monolithic storage array needing to be copied else-where for whatever reason.
 
Last edited:
Farkle said:
What's wrong with the Corsair TX850M? It's stable up to it's capacity at 45c, and has decent ripple and noise suppression. Yes, the TX850 v2 would be a superior power supply (Seasonic vs CWT), but for the load purposes even up to 24 disks on an i3, I don't think it's going to matter.
Click to expand...

You pretty much answered your own question: The TX850M is inferior to the TX850 V2. Hence why I said it isn't that great. With that said, you are correct that that TX850M is solid enough for that setup. But IMO, it's really not worth buying unless you really need a few modular cables.

Considering that the OP would be just fine with a 750W PSU if he's planning on 20+ drives, I'd rather recommend to the OP to get a higher quality PSU rather than a larger PSU.
 
Quesiton, if just backup and simple file serving...why not a HP microserver (they are $200 right now)? This would be a good way to start with low insertion costs. As you learn more, you could go discrete.
 
Are you buying this hardware new? If you want the benefits of ZFS, and you are buying new hardware, you really need to go to ECC RAM. Otherwise, you'll still lose out on a lot of protection offered by ZFS.

Trepidati0n said:
Quesiton, if just backup and simple file serving...why not a HP microserver (they are $200 right now)? This would be a good way to start with low insertion costs. As you learn more, you could go discrete.
Click to expand...

I think this is a good suggestion, hard to beat for the money and it will work for home use--and support ECC.
 
invade said:
Hi all,

First post, but I've been a lurker around just never got to making an account on the forums. One thing though, I admire most of what you all have in terms of builds hella nice rigs hopefully I can reach those ranks at some point. Anyhow! I plan on making a fileserver and then I'll have crashplan run overnight to backup.

Parts
Case: Fractal Design Define XL Titanium E-ATX Or. Lian Li PC-A77FB
PSU: Corsair Enthusiast Series TX850M
RAM: Corsair CMZ16GX3M4A1600C9 Vengeance 16GB
CPU: Intel Core i3 550 Dual Core Processor Clarkdale
MOBO: Gigabyte Z68MA-D2H-B3 mATX LGA1155 Z68
Raid: LSI 9265-8i going raidz2
Bay: Supermicro Black 3X5.25 to 5X3.5IN SATA Drive Bay
HD: 4x Western Digital Caviar Black 2TB
OS: FreeBSD

Would these parts be sufficient for a ZFS system? I do plan on adding more drives over time. Please let me know if any of these parts would be an issue or if there's another alternative that I can go for?
Click to expand...

Case: Excellent, very nice one with plenty of room
PSU: Way overkill, SeaSonic M12II 520 (or S12II) will do fine not to mention that they're about half the price. http://www.newegg.com/Product/Product.aspx?Item=N82E16817151093
RAM: Never go with non stock ram, 1600Mhz is overclocking on that platform. Stock 1333Mhz is the way to go, you don't need ECC but it surely doesn't hurt. I run FreeBSD and ZFS at home without ECC and its been working fine and never had corruption even with bad memory. http://www.newegg.com/Product/Product.aspx?Item=N82E16820233192
CPU: i3 will be fine (as others pointed out the model you've selected is incompatible with the motherboard) I would go for and i5-2400 or i5-2500 if you want Intels KVM which is really nice otherwise just go for a i3 CPU.
Mobo: You want Intel LAN, if you want Intel KVM go for a Intel motherboard with a Q67-chipset if you don't need it just go for a i3 CPU and a Intel H67-chipset motherboard. Intel works a lot better than Realtek especially in FreeBSD.
RAID-controller: Get a M1015 and crossflash it or LSI 9211-8i. That one you've listed is just a waste of money. Keep in mind that it there's no guarantee that it'll work but a reflashed M1015 works fine in my Intel GD45ID motheboard at least.
Bays: Supermicro most likely wont fit due to lack of rails on the sides, you have small metal ones for each 5.25" slot. http://www.newegg.com/Product/Product.aspx?Item=N82E16817994113 should be a good fit.
HDDs: I would go for Hitachi 5K3000-series or newer

//Danne
 
Again, thank you all for the advice!

Bays: Supermicro most likely wont fit due to lack of rails on the sides, you have small metal ones for each 5.25" slot. http://www.newegg.com/Product/Produc...82E16817994113 should be a good fit.
Click to expand...

Hi diizy regards to the supermicro is it not compatible with the Lian-Li or was it not compatible with the Fractal? Also thanks for the heads up for compatibility!

My recommendation would be for an ECC build rather than a desktop build. Though ZFS is end to end CRC, it doesn't validate the input data from memory, so there is still a chance for data loss if the source data has been flipped to begin with. Have you considered AMD? Unbuffered ECC and a cheap AMD build is going to be sufficient for ZFS on FreeBSD or Solaris for a 730MB/s throughput target. You won't even need an SSD ZIL at 16GB of RAM. I'd also consider 5 disks over 4 for raidz2 to offset the parity space loss. With 4 disks, you're getting the same disk space impact as a mirror.
Click to expand...

Farkle, how much more would it cost to get a ECC build? I'm willing to consider. as well for others is it really worthwhile to go ECC? opposed to normal ram?

Quesiton, if just backup and simple file serving...why not a HP microserver (they are $200 right now)? This would be a good way to start with low insertion costs. As you learn more, you could go discrete.
Click to expand...
Trepidati0n In a way it's not a bad idea, though I'm looking for future expansion later down the road i.e as drives more full I'll be adding more as I go.

New build:

RAM: Corsair XMS CMX8GX3M2A1333C9
PSU: Corsair Professional HX750W
CPU: Intel Core i3 2120
MOBO: Intel BOXDH67BLB3 H67
RAID: LSI MegaRAID Internal Low-Power SATA/SAS 9240-8i 6Gb/s PCI-Express 2.0 RAID Controller Card, Kit
CASE: Fractal Design Define XL Titanium E-ATX or. Lian Li PC-A77FB
HD: Hitachi 5K3000 2TB
OS: FreeBSD
BAY: ?
 
Last edited:
RAM: Value Select have the same performance and are cheaper, also XMS3 doesn't follow JEDEC standards if they require 1.65V and that's a bad thing.
PSU: Still way overkill but oh well...
RAID/Controller: Another cheaper alternative that'll work just as good is to get two ASMedia ASM1061-controllers which are about 30$ each and you can later on add an additional 8x PCIe-controller
Case/Bays: Well, its no the case that's wrong its just that Supermicro didn't bother designing it for 95% of all cases available. I'm not sure what they're called but you have these metal holders/flaps at each 5.25" bay that most likely will get in the way.
inside_front_view.JPG

If you look at these bays you'll see that there's a cut out which as far as I can tell the Supermicro ones doesn't have.
1376_g.jpg


//Danne
 
invade said:
is it really worthwhile to go ECC? opposed to normal ram?
Click to expand...

Yes, you need ECC RAM, especially if you are buying new components. I read an .edu-created white paper on ZFS and ECC/Non-ECC, which basically stated Non-ECC with ZFS is a no-go.

ZFS will be able to still checksum the data on the drive, but it will not be able to determine what it gets from RAM is incorrect. ZFS will be telling you your data is good, when there's really no telling because it will checksum the bad data from RAM and report that it's good data.

People run external SATA cases with the Microservers. How much expandability will you need? For my personal use, I've pretty much determined that drive capacity doesn't really outpace my storage needs, and 4x the latest drives is sufficient, generally.

You could also go with previous generation gear, to save money. I upgraded my AMD-based server to ECC for less than $100.
 
invade said:
Farkle, how much more would it cost to get a ECC build? I'm willing to consider. as well for others is it really worthwhile to go ECC? opposed to normal ram?
Click to expand...

ECC isn't that bad for price. Just use any dual or quad core AMD CPU (I have 620 Propus's in production, but I haven't upgraded since late 2009) and any board with an SB710 or SB750 should work. I haven't really looked at DDR3 boards for file server purposes, but for DDR2 ECC I found Asus M4A78-E to be fully Solaris compatible and dirt cheap at the time. FreeBSD will have higher hardware compatibility, so I wouldn't worry about it too much. Pay attention to motherboard manuals of products you research to make sure there is an ECC scrub portion in the BIOS and you'll be fine. As far as I know, almost all AMD CPUs (desktop and Opteron) support ECC on their memory controller, and the board BIOS just needs an ECC stub to fully utilize it. I believe the recommended i3 2120 also supports unbuffered ECC, if you want to stay Intel. I just personally prefer AMD for entry-level file servers that do nothing but generate I/O and maybe run a virtual machine or two. I prefer Intel for anything enterprise-level, though.

Anyone running a production ZFS system with anecdote that they've been running their system for x amount of time without ECC with "no problems", wouldn't even know the problem existed until they went to verify the md5sum of something from a known good source against all of their files stored on their pool, or their file server completely locked up during a write. Not to say they have a problem, but it's not worth the risk for any type of backup or critical data. Such silent corruption can bite you in the ass pretty bad a few months or years down the line, even your backups of your backup server would be corrupt in this scenario (because the source data had a bit flip.) Just Google for discussions on the matter and you'll quickly come to your own conclusions.

This is probably the academic link that was mentioned in this thread: http://research.cs.wisc.edu/wind/Publications/zfs-corruption-fast10.pdf
 
oraldlight said:
Got a link?
Click to expand...

No, found it while Googling ECC vs Non-ECC. I may have saved it on my little external drive, I'll see if I can dig it up.

Another reference point would be Google's memory testing, where they used their server bank to test out memory error frequency. It was surprisingly regular, as in they documented a lot of server errors. It's probably about the only known worthwhile study, as they use TONS of hardware. They did a similar test with hard drives.

Technically, the white paper didn't say you *couldn't* use Non-ECC with ZFS, but that it was kind of a wasted effort.
 
Well, if you have bad memory you'll more than likely experience crashes and what not so I don't see the need for it although it certainly doesn't hurt. ZFS may write bad data but you'll see much more odd stuff going on before that.
//Danne
 
hmm, I had bad memory, it horribly failed a memtest, but the system booted and ran fine (for a few weeks) while I tracked down what was causing checksum issues.

two bad sticks of ram, and one bad sata cable.
 
I can assure you that at least FreeBSD will tell you pretty fast when its not feeling well ;-)
//Danne
 
diizzy said:
I can assure you that at least FreeBSD will tell you pretty fast when its not feeling well ;-)
//Danne
Click to expand...

It will, but it's not going to notice a flipped bit in RAM on your wedding, child, or topless Jennifer Anniston pictures. ;)
 
Looks like the difference between non-ecc and ecc isn't that huge after all heh, about $100 ish. Possible to recommend one of the following? or if there's another one out there? as well is this ram ok?
RAM:
Kingston ValueRAM 8GB (2X4GB) DDR3 1333 CL9 240-PIN SDRAM for the other mobo that is.

Mobo:

Tyan
http://www.newegg.ca/Product/Product.aspx?Item=N82E16813151248&Tpk=Tyan S5510G2NR-LE
Supermicro
http://www.newegg.ca/Product/Product.aspx?Item=N82E16813182252
http://www.newegg.ca/Product/Product.aspx?Item=N82E16813182251
 
For "value" series RAM, I've used a lot of Crucial and it's done well for me, and usually what I buy. Kingston would probably be fine. Micron would be another probably good choice, though I haven't used any in quite a while.

I'd probably lean towards Supermicro, but Tyan would probably be fine as well.
 
You must log in or register to reply here.
Back
Top