How to build my 10+TB raid

O

olol

n00b
Joined
Jul 22, 2013
Messages
12
Hi,

I'm about to buy myself a little christmas present, and I have gotten to the point where I'm not really sure of how I should move forward.

I will start off with 5x4TB drives with the goal of 12TB of storage, i.e. Raid6 or RaidZ2.
I have got an server today running a raid6 with 2TB drives that is too small for me as of today so I will buy myself a new chassi probably a supermicro SC936E16-R1200 and start filling it up with 4TB drives.

I am not sure wether I should stick with Ubuntu and run mdadm raid6, go for esxi with some sort of storage OS (freenas?) ontop of it and have my storage and applocation services on different OS.

The big plus that I see with freenas is the ability to speed up the raid with an SSD, I have not seen that for mdadm without a lot of "haxxes".

The minus with freenas is that today I dont have servergrade hardware, so that is no ECC memories which is a requirement for freenas, also I've heard that freenas ontop of esxi is not that great for production environments. My goal here is to get a dependable storage that "just works".

The reason I need Ubuntu or some other linux dist is because I am familiar with it, and also I have more than one application on my current server that I dont think runs on freebsd.

tl;dr
-New server, should I run ESXI with freenas and ubuntu or just plain ubuntu?
-Raid6 vs RaidZ2?
-Can mdadm handle SSD cache?

Br
Olol
 
Well I don't know too much about this, but from what I've been reading, since I'm also planning a freenas server for my family as a plex server and nas, is that Freenas doesn't really benefit from having an ssd, since the write speed is already inherently slower in ZFS, as the priority is data integrity, and its a software raid not hardware.

If you would like a cheap solution with ECC, the AMD FX and a bunch of AMD setups support unbuffered ecc.
That should be easy to find.

If you wanted to make something small I think this is pretty cool:
ASRock Intel Avoton C2750 2.4GHz/DDR3/SATA3/V&2GbE/Mini-ITX Motherboard and CPU Combo C2750D4I COLOR BOX https://www.amazon.com/dp/B00HIDQG6E/ref=cm_sw_r_awd_C2TLub1Q6CYN8

It has 12 SATA ports, ITX, embedded CPU, ecc support

Going along with this case:
Silverstone Tek Premium Mini-ITX DTX Small Form Factor NAS Computer Cases, Black (DS380B) https://www.amazon.com/dp/B00IAELTAI/ref=cm_sw_r_awd_U3TLub1DDQDJ4

It has eight 3.5in hotswap drive bays, plus some internal 2.5in bays.
Pretty cool case
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
Oh, maybe I should mention that I alredy have the basic server, Motherboard, CPU and RAM
 
- If you care about data security, use ZFS - nothing else
- A SSD extends the RAM as read cache. As RAM is much faster than a SSD, first use as much RAM as possible.
Then you can check ARC cache statistics to decide if you need an additional SSD

about ECC
A Ram error during write gives you corrupt data with any filesystem so if you really care about data security,
ECC is a must on any server and even with ZFS you need backups.

In relation to ZFS, ECC is suggested that often, because it is a remaining reason of undetected corrupted data.
As ZFS verifies checksums of every datablock on every read you have more RAM activity and more
writes if problems are detected than with older filesystems without any serious error detection.
So a fault due a RAM error is more probable than with ext or ntfs..

But given that the chance of an undetected RAM error is as likely as a silent data error on disks where disk capacity
is about 1000x RAM size, this problem is a much bigger problem and such errors can only be detected and repaired with ZFS.
So ZFS without ECC offers a higher level of overall data-security than older filesystems with ECC.
If this would not be the case, all forums would be full of people claiming dataloss with ZFS.

about virtualizing a NAS
This is done and suggested by thousands of people on Solaris and OmniOS without problems. You only need to care
that you only virtualize the Storage OS, not the controller and disks. Therefor you must pass-through the controller with disks.
This requires server class hardware with vt-d and HBA controller that work with ESXi like the LSI HBAs.

I can's say why there should be a problem with BSD/ FreeNas with such a virtualisation.
What most people are saying is that Solaris/ OmniOS is faster with NFS and CIFS compared to BSD/FreeNAS.
 
Last edited:
The main purpose of my storage is to archive non critical files, all critical files are saved on multiple locations. eventhough I should consider a ZFS array to prevent my data from being corrupt by time

IF I were to virtualize my NAS I would have an LSI controller that can be passed-through to the VM running freenas, so it gets the "bare-metal" experience.

Ofcourse RAM is always to be preffered to an SSD, but an SSD is a lot cheaper than RAM and still provides read/write speeds exceeding my 1gb/s limitation of the network.

I guess I will have to take a look into solaris/omniOS aswell and consider them both in a virtual environment. Would be easy to separate all services on multiple servers instead of having them all on the same as it is today.

A few pro's and con's on multiple servers, biggest con I can see is that I will have to learn multiple IPs and which service is on which IP, todayh I have pretty much all services on the same IP, and then a few virtual machines with specific tasks
 
olol,

If you are familiar with ubuntu, stick with that, but go for ZFS on linux instead of mdadm, it is very simple:
Code: 
sudo add-apt-repository ppa:zfs-native/stable
sudo apt-get update && sudo apt-get --yes upgrade
sudo apt-get install ubuntu-zfs
 
rsq said:
olol,

If you are familiar with ubuntu, stick with that, but go for ZFS on linux instead of mdadm, it is very simple:
Code: 
sudo add-apt-repository ppa:zfs-native/stable
sudo apt-get update && sudo apt-get --yes upgrade
sudo apt-get install ubuntu-zfs
Click to expand...

Yes, I have seen the zfs on linux initiative, and It's got me interested, however I'm a bit concerned about the stability of this. Is it "safe" to run as a live environment? Haven't read a lot about it, so I'm a bit sceptical.

Does it have the same need for RAM as freenas has recommended, as of today I only have 8GB ram in my server that will house the 5 4TB drives, and that does not add up to 1GB/TB storage that I've heard is much wanted for freenas.

Another question I get from this is wether or not I should upgrade just to get to the ECC ram, because today my ram is non-ecc.

Next after this upgrade (case + hdd) is MB, CPU and RAM but due to budget that will have to wait for autumn or so. I would still benefit from ZFS self-repair with this installation and non-ECC, is that correct? So the files shouldn't be corrupt over time.

Br
Olol
 
I would not care about stability of ZoL.
ZFS remains the best for your data what you can do.

This 1 GB per TB data rule from the BSD folks is not a "you need it to run stable"
but a thumbnail rule for an average fast server.
On OmniOS I calc 1-2 GB for the OS + about 100 MB per TB data used as read cache as absolute minimum.
The more you have the faster is the server as the RAM is used as a readcache.
Some of my boxes have 10 GB RAM or more per TB data, depends on use case.

ECC is a must for any modern server. Without ECC a RAM error can corrupt data with any OS or filesystem.
As ZFS use checksums on every readed datablock where a checksum error (Silent data error from disk or based on a RAM error)
results in a repair from redundancy. Therefor ZFS has a higher RAM activity than older filesystems so the probalility of an error
due a RAM problem exists but is very low like all RAM problems (scale with amount of RAM).

But given that the probability of a silent disk error is about in the same region than a RAM error and
that the disk capacity is 10 TB vs 10 GB RAM, you have a 1000x better chance to repair a problem with ZFS than
to introduce one with ZFS but without ECC.

So the best what you can do is using ZFS with ECC followed by ZFS without ECC.
The worst is using a hardware-raid without BBU on old filesystem without checksums and without ECC.
 
1. Grab USB Drive
2. Buy a copy of UnRAID
3. Install UnRAID to USB Drive
4. Turn on Computer and boot to USB Drive
5. Bask in the ease of setting up your NAS/SAN without hiccups, problems, or configuration issues.
 
I'd like to second Bigdady92.

I recently went back to UnRAID from hardware raid. The recent Betas have boosted performance fantastically, and finally do cache drives right. Right now, my testbed system has all of it's data drives running BTRFS, and those all get checksumed weekly against the parity drive. (user setting)

It's simple, straight forward, and easy to set up. I'm even running 8 hypervisors via NFS from this system. IO isn't what I'd call fantastic, but it's more than good enough for my purposes.

On the sliding scale of "size vs performance" UnRAID wins for size.
Granted, it isn't free, but I've had my license for about 3 years now, and have gotten a ton of use out of it.
 
I went from UnRAID to Hardware RAID to FreeNAS and went back to UnRAID for simplicity. The $40 price tag is NOTHING for the YEARS of reliability i've gotten and the support that UnRAID offers. I thought I lost everything when my USB drive crashed but Tom from UnRAID helped me rebuild it without issue.


I bought 2 drives for $75 or something for 16 drives on each USB drive. It's over kill for sure but the longevity and support of the product combined with the ease of use makes it an easy buy.


Do yourself a favor and try it out, it's free for 3 drives to run it through a few spins and it'll do everything you want it to do especially as we are only talking about 5 drives here.
 
I've heard some scary things about unraid, for example
[1] - unRAID doesn't have any kind of checksum, and it just ignores silent errors. Even worse, if a parity error is detected as result of a silent error in the data, the parity is automatically recomputed, making impossible to recover the silent error, even manually.
Click to expand...
Sure, you could be running BRTFS ontop of it (or so I've heard) and get that fixed, but then again, btrfs is quite new and relativly untested, so I'm not too sure about that. Yes, I'm a bit afraid of new things when it comes to my storage and backup environment. So I try to get my facts stright and not just trust the first forum post I read :)

The reason I run Ubuntu today is because I have a lot of services running on my server alongside with the storage. And I'm not sure that all of my services will run on unraid, I guess I could virtualise them, but that feels like a bit of an waste.

Lot's of dessicions to be made before installing the OS.

I'm a bit dissapointed that none of the alternatives can fulfill all my needs/wants
*Raid6 (All can do this)
*Add one disk at a time (ZoL cannot do this?)
*Run on Ubuntu (Unraid seems to be unable?)
*Fix Silent Errors (Mdadm cannot do this)
*SSD as Cache for storage(Mdadm seems to be able to, but not stable?)

I'm not sure as wether I will need an SSD for cache, but the ability to use one is appealing to me, otherwise I will just have to put IO intense stuff on the system disk, which will most likely be 2x120GB SSD Raid0


The free trial of UnRaid, is it possible to run it on ESXi and try it out with a few smaller vm-disks, or does it have to be installed onto an USB and run bare-metall?

BR
 
olol said:
Yes, I have seen the zfs on linux initiative, and It's got me interested, however I'm a bit concerned about the stability of this. Is it "safe" to run as a live environment? Haven't read a lot about it, so I'm a bit sceptical.

Does it have the same need for RAM as freenas has recommended, as of today I only have 8GB ram in my server that will house the 5 4TB drives, and that does not add up to 1GB/TB storage that I've heard is much wanted for freenas.

Another question I get from this is wether or not I should upgrade just to get to the ECC ram, because today my ram is non-ecc.

Next after this upgrade (case + hdd) is MB, CPU and RAM but due to budget that will have to wait for autumn or so. I would still benefit from ZFS self-repair with this installation and non-ECC, is that correct? So the files shouldn't be corrupt over time.

Br
Olol
Click to expand...

You do not need 1gb ram / TB for ZFS in a home use environment, nowhere near that. Your 8gb will be fine for that drive configuration.

If you're making a storage server, use ECC.

And for stability, I've had my 11TB pool running on ZOL w/ Ubuntu 12.04 LTS server for a couple of years now and have not had a single problem with it. Just set scrubs with a cron job and get your snapshots backed up, it's easily the best thing I did for my home ecosystem in the last five years.
 
olol said:
I've heard some scary things about unraid, for example

[1] - unRAID doesn't have any kind of checksum, and it just ignores silent errors. Even worse, if a parity error is detected as result of a silent error in the data, the parity is automatically recomputed, making impossible to recover the silent error, even manually.
Click to expand...

You are talking about a microcosm of a non-existant problem. I've run UnRAID for years and commercial SAN equipment for large enterprises and BitRot is like a damn boogie man. Everyone is afraid of it, very rarely if ever seen.


Sure, you could be running BRTFS ontop of it (or so I've heard) and get that fixed, but then again, btrfs is quite new and relativly untested, so I'm not too sure about that. Yes, I'm a bit afraid of new things when it comes to my storage and backup environment. So I try to get my facts stright and not just trust the first forum post I read :)
Click to expand...

LimeTech has moved to BTRFS from ReiserFS due to support and more updates coming down the pipeline. Check the UnRAID forums, you have users who are running 3 disks to those of us who are running 10+ without problems. Tom, founder of UnRAID, is doing a great job in squashing those bugs when and if they occur. Data loss has been very few and far between when those new beta's come up. UnRAID 'beta's' are Release Candidates as they are extremely polished and very few bugs are present. It's the wonky NIC and RAID cards that present problems.



The reason I run Ubuntu today is because I have a lot of services running on my server alongside with the storage. And I'm not sure that all of my services will run on unraid, I guess I could virtualise them, but that feels like a bit of an waste.
Click to expand...

And you can run a full blown Debian with UNRAID or use Docker to spin off Virtual Machines or if you want a true OpenSource Hypervisor there's a build with XEN built in.


I'm a bit dissapointed that none of the alternatives can fulfill all my needs/wants
*Raid6 (All can do this)
*Add one disk at a time (ZoL cannot do this?)
*Run on Ubuntu (Unraid seems to be unable?)
*Fix Silent Errors (Mdadm cannot do this)
*SSD as Cache for storage(Mdadm seems to be able to, but not stable?)
Click to expand...

UnRAID can do all of that including your Fix Silent Error by using a package installed inside your install:
http://lime-technology.com/forum/index.php?topic=35226.0

Haven't used it, probably won't, worth a quick read.

The free trial of UnRaid, is it possible to run it on ESXi and try it out with a few smaller vm-disks, or does it have to be installed onto an USB and run bare-metall?

BR
Click to expand...

Yes you can run Unraid inside ESXi without a problem. Many users do that already. Check out their forums and guides.

I'm a big fan of UnRAID with the user community, the support from the business, and that it just works without having to reinvent the damn wheel and bash my head into my keyboard trying to get this package to work over that other one. As I progressed up the corporate ladder I was/am a fulltime Linux Admin. I hack shit to make it work at work, I don't want to do that at home. Which is why I find the easiest/simplest thing that works and throw my money at it to make it happen (HAI APPLE!)
 
Yeah, I can totally see how you want to get an "it just works" environment at home, I myself is running Apple pretty much everything, but my server is where I put energy to make things work, not because I neccesarily need to, but because I feel a sense of accomplishment getting things/services to work with a bit of on hand things. that integrity check of UnRAID seems a bit unstable reading some of the comments, but it's definitly a great thing that's being developed. And next time I'm about to migrate all my data UnRAID might very well be my weapon of choise!

But for now, I think I will do a bit of both, I will go with mdadm raid6 for my "big" array with 4TB drives that will just have media, movies, series and such on it. and then I will make an ZoL array of my 2TB drives that will have my personal backups, images and such that is in a greater need of integrity checking.

By going this way I feel I will still have my large array that I'm familiar with and have the ability to expand it as I need with one drive at a time. Also I will get to lear a bit of ZoL which I find quite interesting. It seems to be the best solution for long time storage of crucial data.

Unraid will be tested out on a virtual machine before I actually make any installations, just so that I know what it is, how it works and to make sure I dont want it for my storage!

I might upgrade to 16GB ram on the server because of the rest of the services I run that are quite RAM intense, I've not been under 100% RAM usage in a loooong time now.
 
olol, Something else to consider:
If you will be experimenting with different storage platforms, make sure you keep your migration paths open.

If you have 5x4 Tb raid6 mdadm running, and it is 90% full, there is no easy way to migrate to ZFS later, unless you have a way to offload the data to different storage...

I had this problem, and it took all the free storage of 4 friends to help me migrate away from mdadm to ZoL. Essentially, I storred every file I had twice on other storage. I also claimed the USB drives of my whole family. Not a fond memory...
 
Yes, I am aware of this, but I think I will stick with mdadm for my "growing storage" for now, because if I go with ZoL I cannot add one disk at a time as I need them, I will need to have all disks in place from the beginning or add at least 3 at a time (RaidZ2) and then I would probably want to add at least 5-6 drives to get any actual storage as well, so it would be quite costly to expand an Zpool as I've understood it.

The ZoL array I will run is just for smaller backups that for now is just a few 100GB and the array will be 4-6TB so that should be plenty enough for the future =)
 
olol said:
I'm a bit dissapointed that none of the alternatives can fulfill all my needs/wants
*Raid6 (All can do this)
Click to expand...

As far as I know, unraid cannot do a dual parity setup which is a huge negative IMO considering the size that disks (8TB) and arrays are growing to these days while error rates remain constant and are no better than what they were with 1TB disks were the largest we had.
 
What is the consensus on Unraid vs Snapraid? I'm still considering a napp-it type install but it may be overkill and overly complex for my needs.

I'm also kind of surprised xpenology doesn't get more recommendations here given how many times Synology hardware gets mentioned for a "just work out of the box" type of a solution. Seems like xpenology gives you that simplicity with the flexibility of picking your own hardware. I must be wrong though given the relatively few times I see it mentioned.
 
You must log in or register to reply here.
Back
Top