• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Best NAS / SAN OS

C

cfdfireman

Weaksauce
Joined
Oct 8, 2009
Messages
104
I am looking at building a NAS / SAN in the near future and would like to know all of the OS's out there for this and the pro's and con's of each as well as hardware for each and it's pro's and con's. I think this would be helpfull to anyone looking for the information as well. Thanks.
 
I'm interested in this discussion

I've used freenas, currently using whs v1. Looking at consolidating some of my systems.
building a new esxi box, and running OI + nappit. and a few VM's

I would go with unraid if it was free.

I really like the ability to grow pools and shrink pools.

all my important data is backed up to the cloud or kept in another location.
 
There are pro's and cons to each. It really depends on your hardware and your intended use.

ZFS is nice. I like both FreeNAS and Napp-it, currently right now I like FreeBSD because its more polished than Napp-It but I am using Napp-It because of the higher level of ZFS and it definatly has the potential to be a key player in the future.
 
Last edited:
Running OpenIndiana 151a with Napp-It right now. Very happy with its performance and stability so far.

My 8 SATA drives in a mirrored vdev array can achieve 633MB/s sequential reads, 348MB/s sequential writes, and 2,100 IOPs.

Running Windows Experience Index from a VM running on those drives gives me a score of 6.8 for the hard drive.
 
Last edited:
MarkL said:
2,100 Random IOPS? What kind of drives are you using?
Click to expand...

WD5000AAKX. It's a single platter 500GB, 4KB drive.

Running IOMeter from within a VM right now with 67% Random reads, 33% Random writes and averaging 1,700 IOPs. There are 10 other VMs running on these drives at the same time.
 
Something I wrote up awhile ago so some of it may be outdated:

Amahi with Greyhole:
Links:
http://wiki.amahi.org/index.php/Greyhole
http://code.google.com/p/greyhole/
http://www.amahi.org/

Pros:
+ Lets you use different size drives and combine them into one drive pool ala WHS's Drive Extender feature.
+ Can set the number of copies of a folder/file that you want stored on other drives. So unlike WHS where only a single copy of that data is stored on a different drive, you set it to where that file/folder can be copied across every single hard drive in your PC. So 5 drives means 5 copies of your data
+ Supposedly you're still able to remove a drive from that Greyhole server and still be able to read the files on another PC without greyhole installed.
+ Free and works with Linux, which is also free

Cons:
- Still very new and experimental. So not all of the bugs may have been fixed or discovered yet.
- GUI is still relatively clunky
- Takes extensive amount of time to recover and repool data across a new drive after a drive failure
- Amahi has a tendency to try to become the default DHCP and DNS cache server for the entire network
- Uses the old “landing drive” system of the older iteration of WHS

Notes:
* Should you delete a file while using Greyhole, all the other copies are deleted as well.
* While the above example recommend Amahi Home Server, you can use Greyhole in virtually any Linux based OS

Flexraid Basic:
Links:
http://en.wikipedia.org/wiki/FlexRAID
http://www.openegg.org/

Pros:
+ Uses data based parity in which parity data is only kept of the file actually stored, not like with traditional RAID where parity data is is kept of the entire hard drives, regardless of whether or not you have data on those drives.
+ Allows use of different sized drives as a result of the data based parity setup
+ Free
+ Can be used with either Linux or Windows.

Cons:
- Not realtime parity so if the data needs to be protected at all times, not a good idea to use FlexRAID
- The FlexRAID needs periods of time of no usage at all in order to re-sync the RAID
- If multiple users are editing stored files, flexRAID is not recommended

Notes:
* FlexRAID Live does not have the cons of the FlexRAID Base. However it has not been released yet and even it is, it would still be a bit experimental at that stage.
* A little old but still good reading:
http://www.overclock.net/htpc/694948-has-anyone-tried-flexraid-their-media.html

Unraid:
Links
http://www.lime-technology.com/
http://lime-technology.com/wiki/index.php?title=UnRAID_Wiki[/QUOTE]

Pros:
+ Allows use of different sized drives.
+ Should one or multiple drives die, only the data on those dead drives are lost.
+ If the data on the hard drive isn’t being accessed, the hard drive is spun down until needed.

Cons:
- unRAID Free has a limit of 3 drives (2 data, 1 parity).
- unRAID Plus has a limit of 7 drives (5 data, 1 parity, 1 cache)
- unRAID Pro has a limit of 21 drives (19 data, 1 parity, 1 cache)
- Costs $120 for unRAID Pro (21 drives) and $69 for unRAID Plus (7 drives)
- Relatively low write speeds without that cache drive.

Notes:
* “unRAID™ is similar to RAID-4 in that for every n hard drives, there are n-1 data drives, and a single fixed parity drive” - From unRAID website

Linux MDADM Software RAID:
Links:
How to create RAID using Ubuntu Software RAID
Linux Software RAID
MDADM
The Software-RAID HOWTO

ZFS File Server:
Links:
napp-it ZFS server appliance
Building your own ZFS fileserver
ZFSguru NAS fileserver project

Pros:
+ Free
+ Fairly resilient against crashes, corruption, and such due to the copy-on-write feature as well as the snapshots feature.
+ Maintenance free in that the file system will periodically check and fix itself
+ Very flexible in terms of hardware and software changes.
+ Supports encryption of data

Cons:
- May be difficult to learn for a person who've never worked with BSD before
- Requires a large amount of RAM (4GB+ recommended)
- Cannot expand an existing RAID array with additional drives.

Notes:
* ZFS uses a different method of storage expansion:
sub.mesa said:
What you cannot do, is expand an existing RAID-Z (RAID5) or RAID-Z2 (RAID6) array with one or more disks.
But, you can add new disks or RAIDs to an existing pool. So if you have a 4-disk RAID-Z, you can add another 4-disk RAID-Z so you have 8 disks. The second array would share free space with the first; in essence it would be a RAID0 of two RAID5 arrays. ZFS can expand this way.

What you can do, is expanding mirrors and RAID0's. In the example above that's what actually happened: a new array is RAID0-ed with the existing array. New created files will be written to both devices, for additional speed. Setting copies=2 would make files in that directory be stored on both RAID arrays; for extra redundancy.
Click to expand...

* Here’s another thread filled with tips and an excellent hardware list for a large sized ZFS server:
http://hardforum.com/showthread.php?t=1572188

There's also WHS 2011, Openfiler, Solaris, Nexenta, and ZFSGuru as well.
 
You can narrow the field down pretty quick
*I'm assuming we want the equivalent of single drive fault tolerance at a minimum

- Bitrot protection [yes/no]
--Yes means OI/Solaris/FreeBSD currently

- Expandability: Do you need to be able to add one drive at a time, or is adding drives in groups of 3,5 or more acceptable. Also do you need to be able to mix and match drive sizes.
-- If you want/need to add drives one at a time then all the ZFS based OS's are out.
-- If you want to mix and match drive sizes you are looking at unRaid, FlexRaid, or a drobo nas.
-- If one drive at a time is fine and you don't need to mix and match than linux software raid is fine.
---In this category you can also slot hardware raid cards, which opens up WHS2011 as a potential


If you make a value call on bitrot protection vs. expandability now - and then determine your expandability requirements, well you pretty much arrive at an OS. I'd stay away from solutions which are in beta, etc. unless you are very comfortable with them, or are just doing this as an experiment.

You also qualify SAN as an access mechanism. If you want fiber channel, etc. type support then that will put some additional restrictions on your OS. I believe all the linux/solaris/*BSD based distros will support iSCSI target mode.
 
Maybe I am misunderstanding you. You can certainly add drives one at a time to a zfs pool. You can also add drives of disparate sizes to a zfs pool.
 
danswartz said:
Maybe I am misunderstanding you. You can certainly add drives one at a time to a zfs pool. You can also add drives of disparate sizes to a zfs pool.
Click to expand...

But you can't expand a RaidZ vdev. E.g., if you build it with 4 drives you can't add a 5th one to the RaidZ. You can always add another set of drives as another RaidZ vdev to the pool. This is one advantage of hardware raids and MDraid - you can expand the raidset.
 
That's why I was asking for a clarification... I understand the limitations, but others might not have... Frex: the post was somewhat raidz-centric - one might be (as I am) running a stripe of mirrors, in which case adding 2 at a time is fine. etc...
 
Last edited:
Danny Bui nice write up!


personally, i've come from using a hardware nas (readynas duo with 2 drives) then to a ubuntu server setup with MDADM for web based gui (4 drives) to SolarisExpress11 + napp-it (6 drives)
i found setting up the software-raid in ubuntu when using a proper gui over the mdadm web gui to be much easier for some reason.
I'm very new to napp-it and sometimes it seems complicated but im sure it will get easier in time

perhaps i just like things as easy as possible :D
 
Another problem with ZFS is the following (I dont know if all raid variants share the same problem? Can anyone confirm?)

Lets say you have a zfs raid consisting of one raidz2 (raid-6) of 6 disks. You use it for a while, and it becomes full to 90%. Now you add another raidz1 of 8 disks. Here comes the problem: all the old data on raidz2 will not be evenly distributed to the new disks. The old data stays on raidz2 disks and you might still have capacity problems because the raidz2 is full to 90%.

Solution: Say you have a filesystem called /Media on the old raidz2. Then you need to copy this filesystem to an external disk and then back again. All newly copied data will be distributed evenly on all disks. This is clumsy, so you eliminate the need of an external disk, and instead do like this: You create a new filesystem called /Media2 and copy all data from /Media to /Media2. That suffices to distribute all old data, evenly to all disks, including the new disks.

Thus, you need to manually distribute the old data to every new disk. It is not done automatically.

Also, ZFS does not have an defrag utility. The solution to this, is written here.
http://en.wikipedia.org/wiki/ZFS#Limitations

The greatest unique advantage that ZFS is alone to have, is written here:
http://en.wikipedia.org/wiki/ZFS#Data_Integrity
 
danswartz said:
Maybe I am misunderstanding you. You can certainly add drives one at a time to a zfs pool. You can also add drives of disparate sizes to a zfs pool.
Click to expand...

With the qualification that we are talking about systems that can survive at least a single drive failure you really can't add drives one at a time - and maintain that level of protection. (Since if you add a single drive to a zpool, and that single drive fails, you have just effectively killed the rest of your zpool).

And yep, most of the post(s) are RAID-Z centric as you point out. Running a pool of mirrors is not something you would normally do unless you know you are going to need the performance - and in that case you probably don't need the guide. At least that was/would have been my rationale. If someone shy's away from ZFS because of a requirement to add 3+ drives at a time I wouldn't expect them to have a requirement for a pool of mirrors.
 
Sorry, I'm not trying to be a PITA here, but a lot of people reading these posts are going to see "you cannot do X" and take it as an impossibility, not a bad idea. Keep in mind also that the comment about killing the rest of the pool is not correct - you will only lose files that have been written onto that drive since it was added. That said, I don't disagree with the basic "bad idea" part, but I know people who will stripe 4 drives in a raid0 because they need speed and space and none of the file are critical, and having them go mdadm or HW raid because they think you can't add another drive for raid0 is a bummer.
 
brutalizer said:
Another problem with ZFS is the following (I dont know if all raid variants share the same problem? Can anyone confirm?)

Lets say you have a zfs raid consisting of one raidz2 (raid-6) of 6 disks. You use it for a while, and it becomes full to 90%. Now you add another raidz1 of 8 disks. Here comes the problem: all the old data on raidz2 will not be evenly distributed to the new disks. The old data stays on raidz2 disks and you might still have capacity problems because the raidz2 is full to 90%.

Solution: Say you have a filesystem called /Media on the old raidz2. Then you need to copy this filesystem to an external disk and then back again. All newly copied data will be distributed evenly on all disks. This is clumsy, so you eliminate the need of an external disk, and instead do like this: You create a new filesystem called /Media2 and copy all data from /Media to /Media2. That suffices to distribute all old data, evenly to all disks, including the new disks.

Thus, you need to manually distribute the old data to every new disk. It is not done automatically.

Also, ZFS does not have an defrag utility. The solution to this, is written here.
http://en.wikipedia.org/wiki/ZFS#Limitations
Click to expand...

You must differentiate between using the new pool-space and having best performance.
If you have a pool from one vdev (Raidset like a Raid-Z2) and fill it up until its full and you add then a second vdev
you can use the new space without thinking about. ZFS cares about. Its true, that the old data remains on the first vdev
while new bits are stored on the second vdev but thats only a performance question.

If you have enough space, ZFS tries to spread all data over all disks to have best performance. But this is mostly not
a real world problem because pool performance is usually better than net-performance. It could be a question if you need
best I/O performance like with databases or ESXi datastores. In such a case, it may be usefull to move data after
pool expansion to not only have more space but also better performance.

Usually you do not need to care about. ZFS is always doing the best in any use cases.
The same with fragmentation. Its mostly not a real problem. You should only avoid to fill up pools over say 80%.
 
danswartz said:
Sorry, I'm not trying to be a PITA here, but a lot of people reading these posts are going to see "you cannot do X" and take it as an impossibility, not a bad idea. Keep in mind also that the comment about killing the rest of the pool is not correct - you will only lose files that have been written onto that drive since it was added. That said, I don't disagree with the basic "bad idea" part, but I know people who will stripe 4 drives in a raid0 because they need speed and space and none of the file are critical, and having them go mdadm or HW raid because they think you can't add another drive for raid0 is a bummer.
Click to expand...

No worries, and I was purposefully being a bit imprecise by making some assumptions about the intent.

I am interested in what you say about adding a single drive vdev to a pool, and then having that vdev fault.

while it makes sense that the other information is actually available (since the previous data had been striped over the other vdev's), does ZFS itself support pulling out the data, or would you have to result to some sort of fancy forensics service? I can't say I have ever tried it, so I don't know what would happen. (For instance say you had 2 raid-z1 5 disk vdev's, then added a 3rd single drive vdev, wrote 2 files to the pool, then removed the single drive vdev - will zfs really transparently let you pull data fother than those 2 files, or will it just not let you mount the pool - marking it as faulted)?
 
If a vdev crashes, then I suspect all your data is lost.

In your case, if you add a single 3rd drive vdev and the drive crashes, then your entire zpool is lost. I am quite sure on this.

Therefore, each vdev need to have full redundancy and be able to sustain a disk loss. Only then can the zpool sustain a disk loss.
 
brutalizer, do you have a cite for this, aside from being 'quite sure'? not being snarky, but all references i've read state that if a drive fails and there are not enough redundant drives, any files which had data on that drive are lost - i've never seen anything at all to indicate the whole pool is dead. although it doesn't say so explicitly, the zfs best practices guide seems to imply otherwise.

http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide
 
Okay, I did some experiments, and if the entire drive is gone, the pool is in fact unusable. I think the confusion was that in the non-redundant drive case, if you have a URE, only the files involved are inaccessible, not the whole pool.
 
Ok, thank you for your experiment. Now we know for sure.

This shows how important it is to only add a vdev group that have redundancy to be able to sustain a disk crash: Never add a vdev group, which consists of only a single disk.
 
brutalizer said:
Ok, thank you for your experiment. Now we know for sure.

This shows how important it is to only add a vdev group that have redundancy to be able to sustain a disk crash: Never add a vdev group, which consists of only a single disk.
Click to expand...


think of a ZFS pool like a chain of Raids.
Like any chain the weakest element determines overall reliability
 
I use Nexenta Community edition (free up to 18TBs) So far I love it. Little difficult to get going, but once you do, its rock solid. I use a combo of CIFIS and ISCSI targets which both provide good speeds considering my server only runs on 4 Samsung 1TB 7200RPM drives, a mobile 1.3GHZ AMD Neo, and 4GB ram. When transferring large mkvs I can max out a 1Gbit Ethernet connection.
 
You must log in or register to reply here.
Back
Top