Inexpensive ZFS or BTRFS build

facesnorth

Limp Gawd
Joined
Feb 5, 2007
Messages
240
I'm looking to create a relatively inexpensive yet reliable storage solution.

I have 4 x SAMSUNG EcoGreen F4 ST2000DL004 2TB 32MB Cache SATA 3.0Gb/s 3.5" Internal Hard Drive Bare Drives that I ordered back in December of 2010 and never used. I know I'm an idiot for waiting this long to build my storage solution which is why I'm looking to just go ahead and do it inexpensively. I realize these drives have very well lost most of their useful life even just sitting there unused. To make matters worse they were all bought on the same day from the same location, so more likely to fail together. I just don't want that $320 to go to waste.

What do you guys think of those Dell cloud server dual xeon systems being sold on ebay? I see them sold for between $150 - $200. Includes a 1U rack mount enclosure with 2 Xeon processors, 16GB of ECC ram and seems like all I need to get going. Any thoughts on this? What should I install the OS on? Should I put it on one of the drives? I was initially thinking I'd use an SSD or an SD card or something for the OS and just use the hard drives for my array. But not sure what to do in this case.

Thanks.
 

iamwhoiamtoday

Limp Gawd
Joined
Oct 29, 2013
Messages
493
Try looking at one of the HP mini-tower servers.

1U servers are cheaper and sleek, but.... they are LOUD. Believe me, I have a fair number of them that I can't turn on for fear of my roomie beating me with a mallet because of their noise.

http://www.amazon.com/HP-ProLiant-MicroServer-Ultra-Server/dp/B00AKWUZ58 or similar usually works pretty well.

OR! Build your own! Just install FreeNAS on a USB stick, and you'll be good to go!
 

facesnorth

Limp Gawd
Joined
Feb 5, 2007
Messages
240
I've heard they are loud. I am planning to stick it in my basement in an equipment rack which will ultimately be framed inside of a utility closet. Although I realize I might *still* hear it I could insulate it if necessary. I'd love if what I get could fit in the rack, though.

If I build my own server, any thoughts on equipment specs to do it on the cheap using my old HD's?
 

facesnorth

Limp Gawd
Joined
Feb 5, 2007
Messages
240
It's hard to compete with $195 shipped for basically a turn-key solution.... especially if I can keep the noise down. I like the idea of using ECC RAM on a zfs or btrfs system. If I'm gonna piece a system together myself I think it's gonna be a more expensive, more serious system. This could get me by for a few years I think. I might just buy one of these 1U rack servers and call it a day. I'll stick a USB key in like you said, can't believe I couldn't think of that.

Any other gotchas you can think of besides the noise?
 

cantalup

Gawd
Joined
Feb 8, 2012
Messages
758
my suggestion

find cheap parts on ebay

99% my parts bought from ebay.
when buying HD or SSD, I always pick enterprise model :D..not so much $$

do you need real server case 3U 4U or 2U?
 

iroc409

[H]ard|Gawd
Joined
Jun 17, 2006
Messages
1,373
You can sometimes get a Dell T20 direct from Dell for your price range now, with 4GB RAM and a G3220. Add another 4GB and you'd be good to go. They are a little quirky, but are fine.

I would also look for a deal on a T110ii. Mine was pretty decent, and I like it better than the T20.
 

diizzy

2[H]4U
Joined
Nov 6, 2008
Messages
2,602
or you could just rip the T20 mobo out and place it a larger chassis ;-)
The Dell T20 is a very good buy at sub 200$ over at Dell.com
 

facesnorth

Limp Gawd
Joined
Feb 5, 2007
Messages
240
I'm looking at the T20. I may just be ignorant on hardware, but I don't see why the G3220 and 4GB is superior to dual Xeon E5430 and 16GB ECC for the same price?
 

iamwhoiamtoday

Limp Gawd
Joined
Oct 29, 2013
Messages
493
I'm looking at the T20. I may just be ignorant on hardware, but I don't see why the G3220 and 4GB is superior to dual Xeon E5430 and 16GB ECC for the same price?

The G3220 can also run ECC memory, and ECC DDR3 is pretty cheap these days over on ebay. (4GB sticks for cheap.)

According to CPUBoss the G3220 is a better chip in both single and multithreaded operations.

The E5430 came out back in 2007 and uses only expensive/hot running DDR2 FB-DIMMS.

The G3220 idles a lot better, and the platform overall uses a hell of a lot less power than the LGA771 board of the E5430, in addition to having much more modern expansion slots. (SATA 6gbps, USB3 PCI-E 3.0, UEFI, much better power idling options)

tl-dr: The G3220 is faster, has a LOT more options, and upgrading it is cheaper (and it isn't a 9+ year old platform.)
 

iroc409

[H]ard|Gawd
Joined
Jun 17, 2006
Messages
1,373
I wouldn't even consider those old Xeons for a file server. iamwhoiamtoday hit all the high points. Hardware failure on something that old is starting to be a consideration. The T20 is a pretty quiet machine (like a desktop), and will be significantly quieter than a 1U hot Xeon.

I would just get the T20, buy another 4GB stick of ECC RAM to go with what it comes with, and a cheap SSD of your choice for the OS (cheap but decent, like SanDisk or Crucial). 8GB of RAM is sufficient for what you're doing, and I agree ECC is a good idea with ZFS.

For reference, my T110ii is a FreeBSD ZFS file server, and only has a Pentium G1820 in it. It is perfectly adequate, and slower than the G3220. I have a SanDisk 128Gb SSD for the OS, and am using 3% of it. I have a T20 as well, running Server 2012... but it's mostly serving as a web-based thumb drive at the moment.
 
Last edited:

facesnorth

Limp Gawd
Joined
Feb 5, 2007
Messages
240
Thanks guys sounds like great advice.

I think I'm gonna go with the T20.

I think I have an older 64GB or 128GB Intel X25-M SSD laying around. If not I'll buy a Samsung 850 EVO. Would not a flash drive be sufficient?

Wouldn't I also need to get 8GB of ECC RAM and remove the RAM that comes with? Would I get the benefits of ECC RAM with ZFS if I kept the 4GB non-ECC that come with it, and added 4GB of ECC?
 

FireBean

Gawd
Joined
Apr 12, 2010
Messages
994
I would also like to touch on the 1U thing. I had just 1, 1u Server once. It was my first server ever when I wanted to start learning more about them and what they can do for me. I was in a room in my basement. I could hear that thing through my floors at night. Wife could hear it too... which means it was shut down nightly :D
 

facesnorth

Limp Gawd
Joined
Feb 5, 2007
Messages
240
I'm ready to get the T20.

Wouldn't I also need to get 8GB of ECC RAM and remove the RAM that comes with? Would I get the benefits of ECC RAM with ZFS if I kept the 4GB non-ECC that come with it, and added 4GB of ECC?
 

CombatChrisNC

[H]ard|Gawd
Joined
Apr 3, 2013
Messages
1,140
For ZFS, I don't know about other OS's, but I understand that FreeNAS does NOT like chips with a FSB. Go for a newer Intel chip.
 

facesnorth

Limp Gawd
Joined
Feb 5, 2007
Messages
240
Oh ok, wow. So you're saying the T20 actually isn't a good choice for ZFS. Can you suggest an alternative in line with the intentions of my original post? Thanks.
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,051
Intel serverchipset, Intel nic, ECC RAM (you should use only ECC with any server):
perfect for a server, with or without ZFS. RAM minimum at least with Solaris is 2 GB,
more gives you a better performance, 4-8 GB is ok.

Only Problem: only 4 sata ports;
HP G8 or ML 10 v2 as another option would give the option for 4 disks + a small bootdisk.
 

0ptional

Don't Trust Your Friends with Your Decanter
Joined
Feb 22, 2003
Messages
5,591
Oh ok, wow. So you're saying the T20 actually isn't a good choice for ZFS. Can you suggest an alternative in line with the intentions of my original post? Thanks.

He was referencing the older Dual Xeon setup you mentioned.

To answer your other question regarding the RAM: You would need to use either *only* ECC RAM or *only* non-ECC RAM. Plan on removing the 4GB stick that comes with it and installing your 8GB ECC kit you buy (4gb x2 for dual-channel benefits).

Depending on the software you load on the machine, a flash drive for boot might be just fine, I use one on my server for ESXi 6.0, but based on your responses thus far, FreeNAS is perfectly happy on a flash drive and use the 4 internal sata headers for your spinning HDD's.
 

cantalup

Gawd
Joined
Feb 8, 2012
Messages
758
Oh ok, wow. So you're saying the T20 actually isn't a good choice for ZFS. Can you suggest an alternative in line with the intentions of my original post? Thanks.

trust me on this. T20 is good and better that you need to ad HBA card and expander .

I already transplanted Ts140 (lenovo) to SM 2U and 3U server case (used , bought from ebay) and running since late 2014.
I added:
*bumped to 16G RAM unbuffer ecc ( since I can do that, or using 8G RAM should be ok ( basically added 4G RAM stick))
* install dell H200 HBA card ( you can use equivalent) that flash to IT
* since my server case has already sas2 backplane, so easy pie ... just attach 8087 cables from HBA card to backplane
 

diizzy

2[H]4U
Joined
Nov 6, 2008
Messages
2,602
I have a T20 running here in a Factal Design XL case, 2 IBM M1015 HBAs, 12Gb (4+8G) RAM without any issues at all. It comes with 4Gb ECC RAM at least here in Europe. I'm living on the edge so I'm running FreeBSD -CURRENT on mine (FreeBSD 11.0-CURRENT #0 r293117) ;-)
byhve also works fine if you need VMs...

Very useful information here:
http://www.hardwareluxx.de/community/f101/dell-poweredge-t20-1031138.html (translate using Google/Bing etc if you don't read german).
 

iroc409

[H]ard|Gawd
Joined
Jun 17, 2006
Messages
1,373
I would not remove the RAM it comes with, just buy another 4GB stick of ECC RAM.

If you want a part number, go to Crucial's website and go with the "guaranteed" part number. I think that's what I did. No sense in wasting the 4GB it comes with.
 

FireBean

Gawd
Joined
Apr 12, 2010
Messages
994
Also... one thing you could do if you need more drives in the future. Get an External port HBA and a hellua cheap Rackables encloser and go to town. :D
 

facesnorth

Limp Gawd
Joined
Feb 5, 2007
Messages
240
Oh wow thanks for all the responses guys. I've been sick and I just ordered the T20 tonight. I totally would have just grabbed that 32GB ECC if it were still available but looks like I missed it.

I get conflicting advice on mixing ECC with non-ECC. If I end up just buying 4x2 tho, then you guys would put them in slot 1 and 2, and move the non-ECC to slot 3?

Also, still deciding on either a flash drive or an SSD for the OS drive. Should I use this for the log files also? What's an ideal size? If I go SSD would I just get a USB 2.0 external enclosure? USB stick seems simpler and cheaper if there's no major performance hit.
 

diizzy

2[H]4U
Joined
Nov 6, 2008
Messages
2,602
No, you can't mix ECC and non-ECC... Either you have it or not.
FWIW that eBay memory wouldn't have worked either as the mobo doesn't do registered memory and it's too slow (1333 is what the G3220 CPU runs at).

Just grab that Crucial memory I linked to earlier and call it a day. No sane person gets 4Gb sticks nowdays since they limit future upgrades too much. I do however find it quite amusing that you're debating about this and seriously thinking of getting a SSD for the OS (which is overkill by far).

I have no idea what you're going to run but I would highly recommend you to run a full OS such as FreeBSD (perferably -HEAD as it comes with much appareciated enhancements). Running off a USB stick is only viable if you have a distribution that loads more or less everything into memory, this is common for firewall distributions but not so much for storage distros so I wouldn't go in that direction.

You also have another issue, you're missing at least one SATA port if you want a drive for the OS (recommended). Since money seems to be an issue go for a ASMedia ASM1061 PCIe card which supports AHCI meaning that it works with "everything" in terms of OS support.
http://www.amazon.com/IO-Crest-Port...e=UTF8&qid=1452849029&sr=8-4&keywords=ASM1061
http://www.newegg.com/Product/Product.aspx?Item=9SIA1JM0FB8308

As for OS drive, if you want something safe I'd highly advice you to go with something that uses either a Marvell (or Toshiba as they're rebrands), Intel (genuine) or Samsung (non EVO drives). Anything else except Data Center SSDs are more or less "I feel lucky"-mode. These SSDs range from ~50+ and you wont notice a difference in a storage server compared to a regular HDD.

In your case just go for a regular 2.5" HDD and put it where the ODD slot is.
http://www.newegg.com/Product/Product.aspx?Item=1Z4-000B-00003&ignorebbr=1 (40$)
http://www.newegg.com/Product/Product.aspx?Item=N82E16822149377&ignorebbr=1 (50$)
http://www.amazon.com/Toshiba-MQ01A..._feature_keywords_three_browse-bin:4990425011 (44$)
http://www.amazon.com/Toshiba-5400r..._feature_keywords_three_browse-bin:4990425011 (49$)
http://www.amazon.com/Toshiba-5400R..._feature_keywords_three_browse-bin:4990425011 (50$)

They're all overkill in terms of size but they're decent drives and cheap (again, pricing) and if you go for the larger one you can use it as a backup for your most important files from your array.

Just a few tips from your fellow T20 owner...
 
Last edited:

JesDer

Weaksauce
Joined
Jul 9, 2004
Messages
93
I agree with Dizzy as well.



I do believe there is/was cases were you can mix ECC and non-ECC. However, doing so means you would be operating without ECC.

There are also some cases where you can use ECC memory in a system but the memory controller does not support it so you don't get the benefit of ECC. (Check Intel ARK to see if the CPU supports ECC.. the G3220 does).
 

cantalup

Gawd
Joined
Feb 8, 2012
Messages
758
I agree with Dizzy as well.



I do believe there is/was cases were you can mix ECC and non-ECC. However, doing so means you would be operating without ECC.

There are also some cases where you can use ECC memory in a system but the memory controller does not support it so you don't get the benefit of ECC. (Check Intel ARK to see if the CPU supports ECC.. the G3220 does).

Not old i3, celeron Are support ecc udimm.

The question is on motherboard bios. Do the bios disable it or not
 

cantalup

Gawd
Joined
Feb 8, 2012
Messages
758
Buy used enterprise parts on eBay.
You will save $$$$$ than buying new or on newegg.

You need hba card when adding 8 or more drives.

Do not buy asm sata card. Crappy controller.
Get dell h200 or 310. Or IBM m1015.
And flash with it firmware...


Intel ssd is my suggestion.
Buy used s3500 , s3610 Intel on eBay.
They are cheap and reliable. 100g size or less should be OK for os.

I bought 30 bucks for 80gb used Intel ssd sata3 . early last year for os hmm proxmox actually.
 
Last edited:

diizzy

2[H]4U
Joined
Nov 6, 2008
Messages
2,602
The ASMedia controllers are decent and a M1015 will set you back 100+ incl cables, not to mentioned be useless since you can't accommodate more drives in that case. Buying a used SSD without any warranty doesn't seem like a wise choice for saving ~10-20$ bucks.
 

ToddW2

2[H]4U
Joined
Nov 8, 2004
Messages
4,018
The ASMedia controllers are decent and a M1015 will set you back 100+ incl cables, not to mentioned be useless since you can't accommodate more drives in that case. Buying a used SSD without any warranty doesn't seem like a wise choice for saving ~10-20$ bucks.

Sorry but you're just wrong in this post.

There are M1015 listed here and on ebay recently for <100 some down to 75, the cable(s) run 3-8$/each, and depending no the controller 'onboard' the m1015 may be a worthy upgrade/improvement.

Also, used enterprise drives run damn near 50% (and if you watch even more than 50%) off most of the time on ebay not 10-20$ off only... with ebay/paypal you have a built-in return for DOA and drive not as described.
 

diizzy

2[H]4U
Joined
Nov 6, 2008
Messages
2,602
So here's the deal...
http://www.ebay.com/itm/IBM-ServeRA...792837?hash=item3d0e0dd605:g:aeEAAOSw3d1TzaOs
http://www.ebay.com/itm/IBM-LSI-Ser...398873?hash=item1a07f13899:g:qm0AAOSwVL1V-uOC
http://www.ebay.com/itm/IBM-SERVERA...447947?hash=item4881a0dacb:g:E6IAAOSwgQ9Vt8Vf

http://www.serversupply.com/products/part_search/pid_lookup.asp?pid=123862
https://www.google.com/shopping/pro...d=0ahUKEwiOsuix8q3KAhUDjiwKHZdgBdAQ8wII5AUwCA

So, somewhere around 85-120 if you want to buy it from someone in the US, you can save 5 bucks or so buying from some questionable seller in China.

Again buying a decent cable,
https://www.google.com/shopping/pro...BESwKHbALA8oQ2SsICQ&ei=P_2ZVuXbJYGisAGwl4zQDA

Around 15$ per cable unless you want to use some with questionable/unknown quality.

That ads up to above 100+, and in terms of perfomance the Intel AHCI controller is actually faster than the LSI card (and obviously faster than the ASMedia contrller) however it doesn't do any kind of hardware RAID but since we're not going to use that anyway it doesn't matter.

https://www.pugetsystems.com/labs/articles/Product-Roundup-Intel-LSI-RAID-cards-98/
I did find a more recent benchmark somewhere but anyway that shows an old Intel P67 being slightly faster and they've become faster over time if you look at later chipsets (Dell T20 comes with Intel C226). This is also what my IBM M1015 controller performs compared to the Intel controller on me boxes too.

I'm compared the price for used DC drives compared to a new consumer HDD/SSD, again... it's not a data center and money seems to be an issue. Yes, you have DOA but after that you have nothing compared to at least 1y warranty depending on drive (Toshiba seems to have 3y).

Also have in mind that this chassis only fits 6 HDDs in total so I'm going to claim that it's kinda a waste getting a M1015 and better spent on something else.
 

facesnorth

Limp Gawd
Joined
Feb 5, 2007
Messages
240
Alright it looks like I'm buying these 3 items today to complete my server:

http://www.newegg.com/Product/Product.aspx?Item=N82E16820191593
http://www.newegg.com/Product/Product.aspx?Item=9SIA1JM0FB8308
http://www.amazon.com/gp/product/B00AUH3L04

A lot of these choices were suggested due to my sought out price range, which I appreciate. While I am trying to accomplish this build as affordably as possible, do any of you think I am missing an opportunity to do something far smarter by spending only a little more?

As far as why I was considering an SSD, it was because it was suggested to me in one of the earlier posts in this thread.

My intention at the moment is just to run FreeNAS. I am certainly open to doing things differently if there is a beneficial reason to do so. Thanks.
 

cantalup

Gawd
Joined
Feb 8, 2012
Messages
758
Alright it looks like I'm buying these 3 items today to complete my server:

http://www.newegg.com/Product/Product.aspx?Item=N82E16820191593
http://www.newegg.com/Product/Product.aspx?Item=9SIA1JM0FB8308
http://www.amazon.com/gp/product/B00AUH3L04

A lot of these choices were suggested due to my sought out price range, which I appreciate. While I am trying to accomplish this build as affordably as possible, do any of you think I am missing an opportunity to do something far smarter by spending only a little more?

As far as why I was considering an SSD, it was because it was suggested to me in one of the earlier posts in this thread.

My intention at the moment is just to run FreeNAS. I am certainly open to doing things differently if there is a beneficial reason to do so. Thanks.

For good reason
Get hba card on eBay. Cheap..
Lowest was $35

Swing to sth forum. Many wil give your input objectively without break your pocket.

Stay away with crappy sata card..
Buy used entrperise parts and save $$$$$$$$.
Get used ssd enterprise on eBay. A way cheap ...
Endurance is much much more than consumer ssd.
 

facesnorth

Limp Gawd
Joined
Feb 5, 2007
Messages
240
I don't know what sth forum is.

I'm looking for specific suggestions. I wouldn't know which hba card to pick. And a few suggestions made on this thread have turned out not to be good suggestions so I'm looking for things that are clearly consensus.

Not really sure how to determine what's a good or crappy SATA card. Also not sure how to differentiate good from crappy enterprise gear.
 

diizzy

2[H]4U
Joined
Nov 6, 2008
Messages
2,602
@ facesnorth
That'll do just fine, just ignore cantalup in this case...
If you just want to it to serve as a file server FreeNAS is a decent way to go, if you want to learn more give FreeBSD a spin. It's not hard and you'll have a lot more options available.

@ cantalup
Please be more constructive in your replies, we can all do random spamming.
And feel free to prove your point rather than doing "..." and "$$$".
 

facesnorth

Limp Gawd
Joined
Feb 5, 2007
Messages
240
Thanks I want to move along with this so I'm gonna go ahead and do that. I have some (limited) experience with FreeBSD and actually I am trying to switch careers and become a Network Security Engineer using kali linux and such, so I'm going to expect to become more proficient at unix based systems. I'm willing to install FreeBSD but can you give me some direction then on how to set this is up with either zfs or btfs to achieve the goals I'm looking for? (just a file server with preferably 2 drive redundancy). Even if it's just a few links to some good guides or something.

Thanks again
 
Top