Large scale home NAS solution

[teh invincatron]

Hey guys i'm looking at building a big ZFS NAS (10-20 disks) using raid-z2 so I can continually upgrade the disks in it easily without having to reinitialize array whenever I get new disks, and looking for some advice as I think my demands are kind of unusual...

I've seen this Norco storage appliance which is exactly the right form factor that I am looking for but unfortunately it is an appliance not a case, if anyone can recommend some case manufacturers that I should be looking into that would be great. So far I have looked at Norco and Chenbro.

1: What is a case that will support a shitload of drives (at least 10 but not be too deep to fit into an AV rack (I want it to go under my TV with my av receiver/media pc etc) similar to the one above (not to deep also). Hopefully this exists.

2: What have you guys found is the cheapest way to add sata ports? Keep in mind wanting to go atom or equivalent if possible to minimize power so may only have 1 pci-e slot. I realise this would be a big ask so if this can't be done I can change.

3: Performance is not an issue I am happy for it to run slow (slow as in around 100Mb/s).

4: Wanting to go as silent as possible, what is the best way to cool 10+ drives. I'm thinking very large slow fans, is there any other options that I have missed that I should be looking at??

5: I'm wanting to minimize power so possibly looking at using low power CPU's (possibly atoms) - not sure how I would power the disks though possibly a separate custom made power supply?? Has anyone done anything similar to this?

6: I Need to find a good sata port expansion card that is running a chipset that has proven compatibility/reliability under unix. Any recommendations here would be great.

7: Surge protection, I could either disable write caching and run with just a surge protector or I could get an inline UPS and enable write caching. What are your guys opinions on this (I assume that zfs would write cache to memory?, if I'm wrong here please let me know I haven't used zfs before).

 

[dashpuppy]

Hey guys i'm looking at building a big ZFS NAS (10-20 disks) using raid-z2 so I can continually upgrade the disks in it easily without having to reinitialize array whenever I get new disks, and looking for some advice as I think my demands are kind of unusual...

I've seen this Norco storage appliance which is exactly the right form factor that I am looking for but unfortunately it is an appliance not a case, if anyone can recommend some case manufacturers that I should be looking into that would be great. So far I have looked at Norco and Chenbro.

1: What is a case that will support a shitload of drives (at least 10 but not be too deep to fit into an AV rack (I want it to go under my TV with my av receiver/media pc etc) similar to the one above (not to deep also). Hopefully this exists.

2: What have you guys found is the cheapest way to add sata ports? Keep in mind wanting to go atom or equivalent if possible to minimize power so may only have 1 pci-e slot. I realise this would be a big ask so if this can't be done I can change.

3: Performance is not an issue I am happy for it to run slow (slow as in around 100Mb/s).

4: Wanting to go as silent as possible, what is the best way to cool 10+ drives. I'm thinking very large slow fans, is there any other options that I have missed that I should be looking at??

5: I'm wanting to minimize power so possibly looking at using low power CPU's (possibly atoms) - not sure how I would power the disks though possibly a separate custom made power supply?? Has anyone done anything similar to this?

6: I Need to find a good sata port expansion card that is running a chipset that has proven compatibility/reliability under unix. Any recommendations here would be great.

7: Surge protection, I could either disable write caching and run with just a surge protector or I could get an inline UPS and enable write caching. What are your guys opinions on this (I assume that zfs would write cache to memory?, if I'm wrong here please let me know I haven't used zfs before).

How deep is your av rack ?

why 10 drives ? why not 8 ?

Im a huge fan of Norco cases, they are great.

what is your total storage goal ?

Budget ? $$ wise..

 

[teh invincatron]

around 20 inches, but i could make modifications if I need to, no budget at this point.

Norco are great the RPC612b is awesome 17inches deep and holds 13 drives I think but the only problem is that it isn't very aesthetically pleasing, heres a pic

 

[treadstone]

@teh invincatron: I think you have that backwards, Norco and Chenbro are cases not appliances!?!

There are LOTs of builds like the one you are looking for on this forum... just do some searching and lots of reading!

 

[unhappy_mage]

The RPC-4220 has treated me well. If you're doing sata you may want another of the variants that doesn't have SFF-8087 ports on it.

 

[apnar]

so I can continually upgrade the disks in it easily without having to reinitialize array whenever I get new disks

This quote scares me a little bit. Make sure you understand ZFS and how you can expand it. You can't easily add one or two drives, as there isn't a way to expand a raid group. What you can do is add more raid groups, so if you're using RAID-Z2 you'd likely want to add drives in groups of 6 or 10 at a time.

 

[omniscence]

These server cases are built for component density, not for best airflow and tend to be noisy. Even if you tweak the cooling to be very quiet while idle, if put under load there is no way it would not be audible. Add to that the vibrations of 20 harddrives. It is not something I would put in an AV rack.

 

[houkouonchi]

This quote scares me a little bit. Make sure you understand ZFS and how you can expand it. You can't easily add one or two drives, as there isn't a way to expand a raid group. What you can do is add more raid groups, so if you're using RAID-Z2 you'd likely want to add drives in groups of 6 or 10 at a time.

Yeah I was gonna say... ZFS isn't very good for expanding one drive at a time and in that case a hardware raid controller will work better for that IMHO. Also software raid can be quite CPU intensive and you will have extremely lengthy rebuild times with a low end CPU like that so another suggestion for getting a hardware controller (maybe with SAS expander).

I guess the problem with the norco (and almost all other high disk density/capacity chasis) is its depth (at 25 inches).

 

[davros123]

zfs resilvering of a drive under zfs, even with a low end CPU? it's a he'll of alot faster than you might expect, iirc around 70MB/s on my low end amd.

as for expansion, he is talking about swapping the drives out for larger capacity drives, not adding additional drives. this is the way to go. for example, two years ago I bout an array of 1.5 tb drives, 3tb are now cheapish so I swap these one at a time for 3tb drives...no downtime and automatically doubled my pool size.

 

[Blackdragon]

Some thoughts ...

From your questions above.

1. Norco 4220/4020 is your a great solution if you have the space. There isn't too many cheaper option, but if you only need 10 drives max then you can probably get away with a regular ATX case.

2. Might try a lower end RAID controller, but if you want to support 10 drives+ it's not going to be cheap

3. Goes with #2, but 100m/s shouldn't be an issue

4. Not really going to be possible, you cooling 10 drives or more, while you can pickup the 120mm wall for the Norco cases, you're going to need to still push air through all the drives. 'Silent' imo isn't really in the cards with this many drives

5. I would stick with a regular power supply. A Seasonic 560W or 650W (gold) is probably a good option. Atom or SandyBridge (Xeon) is probably a good CPU choice.

6. See my setup in the link below; while not the cheapest solution it leaves you ready for expansion without an issue. Though I'm sure others have cheaper solutions.

7. I'd pickup a UPS, 750/1000VA can be had for around $100-$150 that should keep the power on long enough for most outages.

--

See my thread here, not saying it's the best, but might show you what you might be getting yourself into.

 

[houkouonchi]

zfs resilvering of a drive under zfs, even with a low end CPU? it's a he'll of alot faster than you might expect, iirc around 70MB/s on my low end amd.

as for expansion, he is talking about swapping the drives out for larger capacity drives, not adding additional drives. this is the way to go. for example, two years ago I bout an array of 1.5 tb drives, 3tb are now cheapish so I swap these one at a time for 3tb drives...no downtime and automatically doubled my pool size.

70 MB/sec is pretty damn slow. For a 30TB array that is like a 5 day re-silver time.

 

[bleomycin]

If budget isn't a huge concern i would strongly recommend looking into something from synology, perhaps http://www.synology.com/us/products/DS3611xs/index.php

The biggest plus of going with synology is the ability to expand an array one drive at a time on the fly with varying sized drives, something ZFS and mdadm cannot do.

 

[danswartz]

ZFS can do what you describe.

 

[bleomycin]

ZFS can do what you describe.

Correct me if i'm wrong/my understanding is out dated but your saying you can add additional drives (one at a time, of any size) to a raid-z in zfs to expand it once it has been created? My understanding is if you want to expand a raid-z you have to create an additional array and add it to the pool? With synology you can expand an existing array (raid5,6 whatever) 1 drive at a time. That isn't unique considering mdadm can do this, what is unique is the drive can be of any capacity (adding 3TB drives to an array full of 2TB drives) If ZFS can do this now that is exciting to hear!

 

[smithkt]

Danswartz, I'd like to get a clarification on this as well.

I've seen several members specifically say ZFS will not do what bleomycin has decribed. Such as the quoted post below. If it can, this is a game changer. Most people looking to do single disk expansion probably favor unraid, flexraid, or snapraid.

Yeah I was gonna say... ZFS isn't very good for expanding one drive at a time

 

[danswartz]

I thought we were talking about expanding the pool by replacing drives with bigger ones. I re-read the post I replied to and realized it could be interpreted two different ways. You can, say, pull a 300GB drive and replace it with a 600GB - wait for resilver (if a mirror), then repeat with the other side. But as far as adding drives, while zfs can do it, it is not safe (e.g. raid0). As simple as 'zpool add POOL NEWDISKNAME'. Safer is to add a vdev that is a mirror or a raidz or whatever.

 

[brutalizer]

70 MB/sec is pretty damn slow. For a 30TB array that is like a 5 day re-silver time.

ZFS only resilvers the disk, it does not resilver the entire array. Say the disk is 500GB, but the array is 100TB - then ZFS will only resilver 500GB and no more. Thus, it is quick.

But it is true that resilver of large disks might take days. The other disks becomes stressed and if you encounter a read error (25% of happening), then you have lost all your data. Thus, use raidz2 (raid-6) with large disks.



ZFS can not change the number of disks in a vdev. An ZFS raid consists of several vdevs (groups of disks) each group can be raidz1, raidz2, mirror, raidz3, etc. You can add a new vdev, but never remove a vdev. You can not change the number of disks in a vdev. You can add a new group, that consists of only one disk - which would be dangerous because if that single disk crashes you have lost your raid.

This is true.