Some cliche new build questions

poison5151

n00b
Joined
Apr 24, 2012
Messages
9
Hi all! Long time reader, first time poster here. Thanks to this forum, I've built a 48-drive filer for enterprise use at a 10th of the cost and 3x the performance of commercial solutions. That used Hitachi 2TB drives, RAID10, 2 Supermicro chassis, and hasn't had a single drive failure at a 24/7 workload in 2 years!

Now I'm moving on to conquer the next challenge: a home server. I've picked up a 6U XRackPro and will be using a Norco RPC-4224 as the chassis. Typical usage will be to rip/store Blu-Rays and backup home machines.

My two key decision points are the OS and hardware vs. software raid.

As far as the OS, difficulty of administration is not a concern - I'm a RHCE and pretty sure I can handle Solaris / ZFS / FreeBSD. Things that ARE concerns:

1. Stability. For a while, I will be rolling the dice and not backing up the data elsewhere. I am not very comfortable with software that has many or severe bugs, doesn't have a stable release, is run by one guy a la FlexRAID, has no support, or doesn't have a strong community. I can deal with something very stable that has lacking support.
2. Compatibility. I plan on running several apps that run on multiple platforms (e.g., sabnzbd). I don't want to spend days trying to compile things on a totally obscure platform.
3. Performance. The output of the entire thing will be a massive windows share that I can easily and directly access from the HTPC. I've read that people have had better performance with sharing stuff from Windows natively rather than using Samba on a *nix platform. Either way, my network speed (gigabit LAN) should be the constraint, not the server-side software.

Now, if #1 is fully met, I can throw a 1U Windows server into the rack to accomplish #2 and #3 by installing apps there and hosting the Windows share. However, the server itself will needless to say need a way to access the share on the Norco box, whether iSCSI or some other means.

As far as the RAID decision, I like the flexibility and performance of hardware raid (looking at you ARC-1882i with HP SAS Expander), but it comes with a price tag. Not to mention that it's probably overkill for my usage scenario. My main concern with any solution is that the data is safe (double parity). Also, since I will be buying drives as needed rather than all at once, online array resize is a must.

Believe me, I've read and considered many, many threads on this forum. But the lack of a FAQ / sticky makes it hard to objectively compare solutions and I'm afraid I've got analysis paralysis. So, what do you guys think? What OS and RAID type should I run? What's worked and hasn't for you? All responses are appreciated ...
 
Would this be a deal-breaker with ZFS for you?:
sub.mesa said:
What you cannot do, is expand an existing RAID-Z (RAID5) or RAID-Z2 (RAID6) array with one or more disks.
But, you can add new disks or RAIDs to an existing pool. So if you have a 4-disk RAID-Z, you can add another 4-disk RAID-Z so you have 8 disks. The second array would share free space with the first; in essence it would be a RAID0 of two RAID5 arrays. ZFS can expand this way.

What you can do, is expanding mirrors and RAID0's. In the example above that's what actually happened: a new array is RAID0-ed with the existing array. New created files will be written to both devices, for additional speed. Setting copies=2 would make files in that directory be stored on both RAID arrays; for extra redundancy.
 
If you concern about data security, I would suggest ZFS and Software Raid
(no write hole, expandable, no worry about a controller replacement in a few years,
real data checksums and protection against silent data errors)

You may use ZFSonLinux, but i would use either

FreeBSD due to most ready to use programms (like home media servers)
or something based on Illumos because ZFS development is done there (ex OpenIndiana)
and Solaris has the best SMB server and one of the best iSCSI implementations (imho)

Third option is all-in-one
That is a free ESXi barebone server with several OS on top, storage, media, Linux, Windows etc
with a primary storage VM (OpenIndiana)
 
Last edited:
Would this be a deal-breaker with ZFS for you?:
It's an inconvenience more so than a deal-breaker. I can handle adding drives in groups of 4 or 6. Although, that is what initially turned me off ZFS. I know a lot of people here use it with great success, I suppose I should go read up more on it. How is the performance?

If you concern about data security, I would suggest ZFS and Software Raid
(no write hole, expandable, no worry about a controller replacement in a few years,
real data checksums and protection against silent data errors)

You may use ZFSonLinux, but i would use either

FreeBSD due to most ready to use programms (like home media servers)
or something based on Illumos because ZFS development is done there (ex OpenIndiana)
and Solaris has the best SMB server and one of the best iSCSI implementations (imho)

Third option is all-in-one
That is a free ESXi barebone server with several OS on top, storage, media, Linux, Windows etc
with a primary storage VM (OpenIndiana)
I don't think ZFS on Linux is stable enough for something that I'll be trusting all my data to. I will look into OpenIndiana. Staying away from ESXi due to performance overhead and also trying to keep it simple.

Other thoughts?
 
I will look into OpenIndiana. Staying away from ESXi due to performance overhead and also trying to keep it simple.

Other thoughts?

OpenIndiana is the best for storage on earth, unless you look for a dlna server or similar.
(may change in future). There are better solutions for that. (ex Windows)

On the other side, there is only a very very performance overhead with esxi and modern hardware virtualization (storage server with >95 % of performance compared to a barebone server with same RAM and CPU assignement). Overhead is minimal. ESXi is like a bios, managed by a Windows app. Nothing better for a home server or a lab machine. I use it even for my production systems.
 
OpenIndiana is the best for storage on earth, unless you look for a dlna server or similar.
(may change in future). There are better solutions for that. (ex Windows)

On the other side, there is only a very very performance overhead with esxi and modern hardware virtualization (storage server with >95 % of performance compared to a barebone server with same RAM and CPU assignement). Overhead is minimal. ESXi is like a bios, managed by a Windows app. Nothing better for a home server or a lab machine. I use it even for my production systems.

Running even a storage vm these days is very close to bare metal, and opens up a lot of options, especially for potentially having a more efficient use of resources. It's definitely worth consideration.
 
OpenIndiana is the best for storage on earth, unless you look for a dlna server or similar.
(may change in future). There are better solutions for that. (ex Windows)

On the other side, there is only a very very performance overhead with esxi and modern hardware virtualization (storage server with >95 % of performance compared to a barebone server with same RAM and CPU assignement). Overhead is minimal. ESXi is like a bios, managed by a Windows app. Nothing better for a home server or a lab machine. I use it even for my production systems.

Hmm. That certainly changes some preconceptions I had. I also read through your thread, it was really helpful.

How's this for a hypothetical all-in-one build:
SM X8SI6-F
Intel X3470
8 or 16GB ECC RAM
HP SAS expander

I'd use the SATA ports on the motherboard for 2 small OS drives in RAID1 (EDIT: anyone know where to mount presumably 2.5" OS drives in the RPC-4224?) and dual-link the 2 SAS ports to the HP SAS expander, and connect that with 6 sff-8087 to sff-8087 cables to each of the backplanes in the RPC-4224 case.

I would use the OS drives for ESXi and OpenIndiana storage and pass through the LSI 2008 (in IT mode) to the OpenIndiana VM, with a CIFS/NFS share serving as the bulk storage for the HTPC and additional VMs.

Would that work? How fast would CIFS be between VMs on the same box? Any problems with SAS expander and OpenIndiana? Would this be able to handle raid-z2 with 6 drive arrays with reasonable performance?
 
Hmm. That certainly changes some preconceptions I had. I also read through your thread, it was really helpful.

How's this for a hypothetical all-in-one build:
SM X8SI6-F
Intel X3470
8 or 16GB ECC RAM
HP SAS expander

Should work, by I would use:

-a more modern Sandy Bridge board http://www.supermicro.nl/products/motherboard/Xeon3000/#1155
ex X9SCL+-F due to its two (out of the box) ESXi supported Nics
-expanders with the newer LSI SAS2 chipset like a Intel RES2SV240
but avoid expanders in general when not really needed
in this case, i would use 2 SAS2 controller like IBM 1015 (flashed to IT mode, look at Ebay) or original LSI HBA's
- in any case, use 16 GB (at least 6GB for storage VM)

about the boot disk
I use 40 GB Intel SSD's as boot disks with double sided duct tape to "mount" it inside.
For home use, you do not need to mirror. There is usually nothing really relevant on them.
If it fails, you must reinstall ESXi and OpenIndiana and reimport a datapool - down time less than 45 min
 
Last edited:
Thanks for the recommendations, Gea. I'm going to try an all-in-one build. I pulled the trigger on the following:

Chassis: Norco RPC-4224
PSU: Seasonic X750 Gold
Motherboard: Supermicro X9SCL+-F
RAM: 16GB Kingston DDR3 1333 Unbuffered ECC (2x KVR1333D3E9SK2/8G)
CPU: Xeon E3-1240
OS Drive: WD 7200RPM 160GB 2.5"

Also planning to pick up the following in the near future:

HBA: IBM M1015
Expander: Chenbro CK23601
Storage drives: Hitachi 5K3000

I will most likely go with the expander after all. It's LSI SAS2 based, and has 36 ports. I don't want to buy 3 M1015's and have to either find a motherboard with 3 8x PCIe slots or put 2 in a 8x slot and 1 in a 4x slot. Also, if I ever decide to go with hardware RAID, I can pick up a 4 or 8 port card and still use it with the expander, rather than have to sell the 3 M1015's.

I'll try to keep this thread updated with my progress. Cheers! :D
 
Back
Top