• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Large Scale NAS

L

librarat

n00b
Joined
Jun 20, 2013
Messages
18
tl;dr:

I want to build a 12+TB NAS to replace my aging system (hardware raid 6, 6 1.5TB RE4's). What are your suggestions for this approach?
3u 24bay norco case, or its sas expanding cousin? ZFS? Talk nerdy to me!


===================================


It's be a long time coming, but I'm finally embarking down the "rebuild my nas" road. Currently, I have a 6TB nas that works great - 6x 1.5TB RE4's in RAID6 (hardware)
This box runs Debian 6 and provides Samba (home vLAN) and iSCSI (development vLAN) for my ESXi boxes.

I'm out of space. I would like to double this, or go beyond. I have a 3,500$ budget.

This box will be 10Gig equipped (single PCI-E 10G-SR card). Form-factor isn't paramount, though rackmount is desirable. Data resiliency is also critical.

I love the idea of ZFS. It would cut costs back considerably, too. However, there is no native encryption for the entire zpool in anything other than Solaris. This is very much a con for me.

Conversely, hardware raid with >8 drives gets expensive - quickly. A 24bay 3u chassis that's SAS Expanding is about 1300 (for a good one), vs a 24bay JBOD enclosure for ~400$.

So I have the following questions
  1. What route would you suggest, and why?
  2. ZFS or Hardware RAID?

Encryption of the FS (not root, per se) is paramount. So either the Controller needs to do it, or something akin to LUKS/Native encryption in ZFS is a must.

Use case:
Most IO will be against >400M files, though some will be binary reads though thousands of files that are <256K in size (tl;dr: load it all into ram prior to searching - I'm not concerned about write speed for these).
This is my backup - data integrity is paramount. Given the volume of storage, keeping backups isn't really an option here - yes I can suffer a loss, but it'd hurt. It's nothing I can't replace.
 
Last edited:
You could use FreeBSD and ZFS. FreeBSD has an optional additional layer of encryption called.... GELI?

But that would tie it to FreeBSD. And Solaris ZFS would tie it to Oracle Solaris.
 
brutalizer said:
You could use FreeBSD and ZFS. FreeBSD has an optional additional layer of encryption called.... GELI?

But that would tie it to FreeBSD. And Solaris ZFS would tie it to Oracle Solaris.
Click to expand...

Yep, I get that. I was more wondering if the collective "you" would suggest going ZFS for my build, or stick with a hardware controller. Bitrot is always a concern, but so too is encryption of the underlaying fs.

Where's the tradeoff given my requirements?

12+TB is proving to be a challenge...

Edit: Another option I suppose would be a FUSE driver to encrypt data as it's written/decrypt as it's read... Though this seems overly complicated.
 
12TB is only 3 hard drives. Plus whatever you want for RAID.

A case should cost under $100. Really hard to spend over $1000.
 
GeorgeHR said:
12TB is only 3 hard drives. Plus whatever you want for RAID.

A case should cost under $100. Really hard to spend over $1000.
Click to expand...

Assuming I use 3 drives, yes. That's 3x 4TB drives = 12TB Raw. You're a failure* if you do this. At minimum, that'd actually be 5 drives for raid 6, though with drives that big, you're really looking into doing raid 1+0 for the entire array, so you're immediately at 6 drives.

Given that I have a 10GbE uplink and I'd like to use it, more drives is better. Given that raid contollers have 8 internal connections, that's rather limiting.

So here we are back to the million dollar question. ZFS with HBA's, or sas expanding? I'm open to alternatives, too! :)

* I mean that as a statement, not an insult to you specifically. Given that I mentioned data integrity in the OP, using JBOD with 3 drives isn't an option.
 
Last edited:
librarat said:
So here we are back to the million dollar question. ZFS with HBA's, or sas expanding? I'm open to alternatives, too! :)
.
Click to expand...

Expanders are ok with many disks, external disks or if you can use only one HBA -
but you should use SAS disks

In your case all disks are internal, you use Sata and you can use 3 HBAs (ex quite cheap IBM 1015):
Avoid the expander
 
What would you use for disks for your ZIL?
How do you feel about the Norco 24 bay hotswap case?

Why avoid the expander? Without it, finding a mobo with enough PCI-e slots becomes a challenge.
 
librarat said:
What would you use for disks for your ZIL?
How do you feel about the Norco 24 bay hotswap case?

Why avoid the expander? Without it, finding a mobo with enough PCI-e slots becomes a challenge.
Click to expand...

- Intel S3700 is quite ok for a ZIL
(you only need one for sync write ex ESXi + NFS)
- Norco is ok. Not as solid like SuperMicro or Chenbro but ok
- no one has told ever about problems with SAS disks in contrast to Sata
where you must carefully check disk compatibility and cable length.
In case of problems it may be difficult to find the problem or problem disk as well
(I had such problems with WD Raptors)

- enough slots - no problem with server boards - up from quite cheap uATX ones,
- http://www.supermicro.nl/products/motherboard/
 
Norco 24bay RPC4224 FTW (i use that connected to another box) with 10GbE, no expanders. NOTE though you may get a bad port on one of the backplanes, make sure to order a couple of spares so you don't have to wait for RMA.
 
war9200 said:
Norco 24bay RPC4224 FTW (i use that connected to another box) with 10GbE, no expanders. NOTE though you may get a bad port on one of the backplanes, make sure to order a couple of spares so you don't have to wait for RMA.
Click to expand...

I've been reading horror stories about how feeble this case is (the 4224), but it's look badass... It's also reasonably inexpensive. I assume that you're using it as a JBOD box, and not using the SAS expanding version?

When ordering backups, do you mean of the actual backplane lane, or something else?
 
> When ordering backups, do you mean of the actual backplane lane, or something else?
I mean the actual backplanes there are 6 in those units, sometimes a port (SATA port) will be bad, so its good to order > 1-2 spares.
 
librarat said:
Assuming I use 3 drives, yes. That's 3x 4TB drives = 12TB Raw. You're a failure* if you do this. At minimum, that'd actually be 5 drives for raid 6, though with drives that big, you're really looking into doing raid 1+0 for the entire array, so you're immediately at 6 drives.

* I mean that as a statement, not an insult to you specifically. Given that I mentioned data integrity in the OP, using JBOD with 3 drives isn't an option.
Click to expand...

You mentioned cost twice:

"I love the idea of ZFS. It would cut costs back considerably, too. However, there is no native encryption for the entire zpool in anything other than Solaris. This is very much a con for me.

Conversely, hardware raid with >8 drives gets expensive - quickly. A 24bay 3u chassis that's SAS Expanding is about 1300 (for a good one), vs a 24bay JBOD enclosure for ~400$."

The purpose of my post was to indicate there there were huge savings available on the hardware side.

I will let you go off on your own.
 
librarat said:
tl;dr:

I want to build a 12+TB NAS to replace my aging system (hardware raid 6, 6 1.5TB RE4's). What are your suggestions for this approach?
3u 24bay norco case, or its sas expanding cousin? ZFS? Talk nerdy to me!


===================================


It's be a long time coming, but I'm finally embarking down the "rebuild my nas" road. Currently, I have a 6TB nas that works great - 6x 1.5TB RE4's in RAID6 (hardware)
This box runs Debian 6 and provides Samba (home vLAN) and iSCSI (development vLAN) for my ESXi boxes.

I'm out of space. I would like to double this, or go beyond. I have a 3,500$ budget.

This box will be 10Gig equipped (single PCI-E 10G-SR card). Form-factor isn't paramount, though rackmount is desirable. Data resiliency is also critical.

I love the idea of ZFS. It would cut costs back considerably, too. However, there is no native encryption for the entire zpool in anything other than Solaris. This is very much a con for me.

Conversely, hardware raid with >8 drives gets expensive - quickly. A 24bay 3u chassis that's SAS Expanding is about 1300 (for a good one), vs a 24bay JBOD enclosure for ~400$.

So I have the following questions
  1. What route would you suggest, and why?
  2. ZFS or Hardware RAID?

Encryption of the FS (not root, per se) is paramount. So either the Controller needs to do it, or something akin to LUKS/Native encryption in ZFS is a must.

Use case:
Most IO will be against >400M files, though some will be binary reads though thousands of files that are <256K in size (tl;dr: load it all into ram prior to searching - I'm not concerned about write speed for these).
This is my backup - data integrity is paramount. Given the volume of storage, keeping backups isn't really an option here - yes I can suffer a loss, but it'd hurt. It's nothing I can't replace.
Click to expand...
So the tiny NAS you have been using is a backup target? If so then RAID6 was wasted overkill.
10GbE is nice but unless the new main-board has it, wait till you have rest of system before looking at specific NIC's.
Budget is OK, depending on what currency you are referring to....
ZFS for those who are familiar with it and know how to use it, if not the HW RAID FTW.
Norco 4224 is 4RU, don't know where you keep getting idea of 3RU.


librarat said:
Assuming I use 3 drives, yes. That's 3x 4TB drives = 12TB Raw. You're a failure* if you do this. At minimum, that'd actually be 5 drives for raid 6, though with drives that big, you're really looking into doing raid 1+0 for the entire array, so you're immediately at 6 drives.
Click to expand...
His comment was "Tongue-in-Cheek" and to point out that you don't look at a 4224 for only 12TB of space. 40-60TB is more like it.

librarat said:
I've been reading horror stories about how feeble this case is (the 4224), but it's look badass... It's also reasonably inexpensive. I assume that you're using it as a JBOD box, and not using the SAS expanding version?
Click to expand...
Check the dates of the "Horror Stories", some of those stories might even been mine as I have suffered the issues and also how to fix and repair them myself. Hint, stay away from the green Backplanes if using mini-SAS backplanes.




BUILD:
  • Norco 4224 4RU
  • Decent Main-board with at least quad GbE or single 10GbE and plenty of PCI-E slots - Server boards only, reasons bellow
  • Single Xeon Quad or Hex
  • ECC RAM to suit native slots (eg. Quad DIMM's for s2011)
  • Good single rail 800W+ PSU with good 12V rail current
  • HBA and/or
    • -1x M1015 (or equivalent) LSI IT flashed + Chenbro 36-port SAS expander
    • -or
    • -3x M1015's LSI IT flashed
  • Replace rear and mid fans with slightly more quiet fans if noise is an issue but don't fall for the 120mm fan option. 120mm fans simply DO NOT have enough tractive effort (Static Pressure) to pull enough airflow through the drive bays and end up in cavitation
  • Use with either server 2012 + Storage Spaces for those who like Windows and Hyper-V otherwise, a ZFS based OS.
 
Lost-Benji said:
BUILD:
  • Norco 4224 4RU
  • Decent Main-board with at least quad GbE or single 10GbE and plenty of PCI-E slots - Server boards only, reasons bellow
  • Single Xeon Quad or Hex
  • ECC RAM to suit native slots (eg. Quad DIMM's for s2011)
  • Good single rail 800W+ PSU with good 12V rail current
  • HBA and/or
    • -1x M1015 (or equivalent) LSI IT flashed + Chenbro 36-port SAS expander
    • -or
    • -3x M1015's LSI IT flashed
  • Replace rear and mid fans with slightly more quiet fans if noise is an issue but don't fall for the 120mm fan option. 120mm fans simply DO NOT have enough tractive effort (Static Pressure) to pull enough airflow through the drive bays and end up in cavitation
  • Use with either server 2012 + Storage Spaces for those who like Windows and Hyper-V otherwise, a ZFS based OS.
Click to expand...

Thank you so much for the insight! I've been thinking that the hardware RAID route might be the best way for me at this point, at least until ZFS is more native in Linux (yay license incompatibility? :C ).

I've already go the 10Gig infrastructure, so those costs aren't a part of this build :) I went with two single-port Intel 10G XF SR adapters - one for my primary ESXi machine, one for the NAS (they connect through my switch).

Regarding that, I've been considering this and this together, but given that this box won't be doing anything significantly CPU or RAM intensive, do I really need to go the Server hardware route, or can I get away with using commodity hardware and that raid controller?

Further, sticking with this, a 12 bay case would allow for quite a bit of forward momentum in the future. All I'd need would be a sas expander and I'd be good to go if I require >8 disks. Using 4TB disks with that controller (RE4's), I could get a decent density out of it, too.

The downside I see to this is that by using disks this large, and limiting myself to 8 total (before sas expander), my IO won't be all that significant. So while I'll still be benefiting from >1GbE, 10GbE is just overkill until I get enough disks to start to saturate the total throughput. So maybe we jump straight to a 16 bay Norco case, the sas expander, and 2 or 3 TB disks?

Thoughts?
 
Last edited:
librarat said:
Thank you so much for the insight! I've been thinking that the hardware RAID route might be the best way for me at this point, at least until ZFS is more native in Linux (yay license incompatibility? :C ).
Click to expand...
I am not a Unix/Linux user by any means but I did try using OI + NappIT a while back and it was rather good for what it was and I feel it would be wise to keep it as a serious contender considering the money about to be spent and the likely explosion of data capacity you will be wanting. Windows and HW RAID would be my choice because it's where I play around most but my next upgrade will see a Dell 1950 Gen-III with 32GB FBDIMM and twin quad-cores to drive quad-port or 10GbE NIC as well as a M1015 (LSI IT flashed) on OI + NappIT.

librarat said:
I've already go the 10Gig infrastructure, so those costs aren't a part of this build :) I went with two single-port Intel 10G XF SR adapters - one for my primary ESXi machine, one for the NAS (they connect through my switch).
Click to expand...
If you have them, then sweet. I would aim for twin-ports in future but that's for another day.

librarat said:
Regarding that, I've been considering this and this together, but given that this box won't be doing anything significantly CPU or RAM intensive, do I really need to go the Server hardware route, or can I get away with using commodity hardware and that raid controller?
Click to expand...
http://www.lsi.com/products/raid-controllers/pages/megaraid-sas-9266-8i.aspx#tab/tab2
The RAID cards you are looking at don't get me excited at all and I would be betting they will not work well with any parity based RAID (RAID 5, 6, 50, 60) as they lack cache and the balls from a quick look. I will leave them for others who have them to confirm/deny my thoughts on them.

If you want to go ZFS, hardware should be good and have some balls to spare as all of the work is done on the system rather than a HW card. Server hardware doesn't cost that much these days and the added ability to use the spre resources of bigger gear for VM's comes in handy for those who don't want multiple systems running all the time or the space.

librarat said:
Further, sticking with this, a 12 bay case would allow for quite a bit of forward momentum in the future. All I'd need would be a sas expander and I'd be good to go if I require >8 disks. Using 4TB disks with that controller (RE4's), I could get a decent density out of it, too.
Click to expand...
The cost of the 12 Bay is in most cases not going to save you too much. The Norco 4224 (and the knock-offs) still give best space for cost and when backed with a Expander, drive count porn is on the cards for whenever you want more space.

If you want to put system in same chassis then here are the options:

If you want just space and keep the main system on another chassis then here are options:
As you see, some have redundant PSU's and/or built-in expanders.

librarat said:
The downside I see to this is that by using disks this large, and limiting myself to 8 total (before sas expander), my IO won't be all that significant. So while I'll still be benefiting from >1GbE, 10GbE is just overkill until I get enough disks to start to saturate the total throughput. So maybe we jump straight to a 16 bay Norco case, the sas expander, and 2 or 3 TB disks?
Thoughts?
Click to expand...
I think for what you first mentioned about the system being a backup target rather than a primary data server, you don't need huge requirements and certainly not 10GbE. You haven't mentioned the full scope of the what you do with the greater systems (what needs this backup) and given the full picture to make more recommendations.
ZFS and decent drive pool sizes will see good use of the 10GbE but if only want to use it a backup target/dump then I wouldn't even bother with huge or complicated setups. Just JBOD and drives in span will do fine and stick with a couple of 1GbE NIC's teamed.
You did mention briefly that you backup other machines but also as an iSCSI host so maybe give some more though here as to what works well with iSCSI.
 
Last edited:
OP, If you are thinking of the Norco 24bay server, there is an alternative that a lot of folks have done (myself included): search for Supermicro SC846TQ on eBay. There is a seller - TAMSolutions that are selling these systems for a little under $400 shipped. The guts of the system aren't the best, but you can upgrade the AMD systems to dual 6-core procs. These cases are jet engine loud, so you will need to invest in some new fans and a lot of folks are pulling out the redundant power supplies and replacing with a standard ATX PSU (this is for the noise).
The build quality on the case is really good. Here are some links to get you started if this is a direction you want to go.

Personally I am going to give FlexRaid a go. I've got a 24x 4TB server setup right now. Look into FlexRaid if non-hardware raid is of interest (there is also a freeware equivalent called SnapRaid).

http://www.avsforum.com/t/1412640/are-you-looking-for-a-less-expensive-norco-4220-4224-alternative/0_100 This is the most comprehensive info about the cases that I've found, but it does discuss several different cases.
http://blogs.dootdoot.com/mike/2013/09/24/sc846tq-modification-part-1/

I hope this helps!
 
As an eBay Associate, HardForum may earn from qualifying purchases.
vorpel said:
OP, If you are thinking of the Norco 24bay server, there is an alternative that a lot of folks have done (myself included): search for Supermicro SC846TQ on eBay. There is a seller - TAMSolutions that are selling these systems for a little under $400 shipped. The guts of the system aren't the best, but you can upgrade the AMD systems to dual 6-core procs. These cases are jet engine loud, so you will need to invest in some new fans and a lot of folks are pulling out the redundant power supplies and replacing with a standard ATX PSU (this is for the noise).
The build quality on the case is really good. Here are some links to get you started if this is a direction you want to go.

Personally I am going to give FlexRaid a go. I've got a 24x 4TB server setup right now. Look into FlexRaid if non-hardware raid is of interest (there is also a freeware equivalent called SnapRaid).

http://www.avsforum.com/t/1412640/are-you-looking-for-a-less-expensive-norco-4220-4224-alternative/0_100 This is the most comprehensive info about the cases that I've found, but it does discuss several different cases.
http://blogs.dootdoot.com/mike/2013/09/24/sc846tq-modification-part-1/

I hope this helps!
Click to expand...

This looks interesting - the 16 bay SuperMicro case I've been eying is about 200$ more than this, bare. Regarding something Lost-Benji said, and something I hadn't really considered.. was using this new build as a my VM box as well as my data pile... Given that, is this case upgradeable to other, more modern hardware? Say, a dual 2011 board?

Lost-Benji said:
The RAID cards you are looking at don't get me excited at all and I would be betting they will not work well with any parity based RAID (RAID 5, 6, 50, 60) as they lack cache and the balls from a quick look. I will leave them for others who have them to confirm/deny my thoughts on them.
Click to expand...

Which cards would you recommend?
 
Shockey said:
http://www.ebay.com/itm/Supermicro-...311140?pt=COMP_EN_Servers&hash=item27db150024

:confused:
Click to expand...
Still cant reproduce the search..................



librarat said:
Given that, is this case upgradeable to other, more modern hardware? Say, a dual 2011 board?
Click to expand...
Most chassis will and do support CEB/SSI formfactors and will take server boards. The only boards that won't fit are the HPTX monsters.


librarat said:
Which cards would you recommend?
Click to expand...
Ah well, now this is where you will have to spend a lot of time doing homework. There is no one card that will tick all the boxes. You need to look at the drives you have and intend on using with said card then checking to see if they are compatible. Plenty crads out there that only work with enterprise drives and bork at domestic drives.

That being said, I have successfully used Intel (rebadged LSI) RAID cards with both SSD's and ST3000DM001's without fault. I also am aware that plenty use the Areca cards with great success. My personal taste is for LSI & Areca, but as I run Server 2012 & Storage spaces, I don't use HW in my servers.

Ask around and read up as you need to be 100% before outlaying big coins.
 
Last edited:
As an eBay Associate, HardForum may earn from qualifying purchases.
You must log in or register to reply here.
Back
Top