Storage Server Options

E

ehcah

n00b
Joined
Nov 10, 2010
Messages
7
Hello,

I currently run an unRAID, 12TB usable NAS in an external 8 bay enclosure attached to one of my servers via a port multiplier card. To say the least, performance is slow. A lot is most likely due to the cheap card that attaches the enclosure. The rest is mostly as a result of how unRAID writes data. I should point out that we have 3 dedicated HTPC's in our house and I have never had any stuttering or quality issues.

Having said that... I have purchased a 24 bay 4U Server Chassis, MOBO, i5 Processor, 8GB of RAM, SSD drive for OS or Cache, alternate USB thumb drive for OS (solution dependent) and an Areca 1280ML 24 port controller. Once the hardware arrives, I will populate 12 of the 24 bays with 2TB drives. I will NOT rebuild with unRAID and I want to move while I easily have enough empty disk to move data. My gut is telling me there are other, more scalable solutions out there. I'm probably thinking a Raid 6 like solution and between 18TB-20TB usable out of the 12 x 2TB drives. Management, stability/reliability, ease of expansion and future drive replacement when larger drives become affordable are my biggest considerations.

Rather than asking what I should do, I'd like to solicit users feedback on the following solutions. If you feel like pointing out pro's/con's, great. I'm mostly trying to figure out which of these solutions is most popular with the home media crowd.

My short list includes, FreeNAS, Nexenta or simply building another Ubuntu Server using Raid 6, LVM and samba shares. ZFS is my target with two of the solutions, the last is probably just plain old faster, but a full server install rather than appliance. I have ruled out unRAID, EON, OpenFiler, Debian + zpool, OpenIndiana and Windows Home Server.

For my particular application, storage focus is for Movies, Music, Pictures and a segregated area for backing up our home pc's. I should note that some of the solutions I've ruled out are based on gut feel and/or my experience testing them in a VM. I do not feel that any of them are bad solutions.

Thank you for any time consideration given to my post.

ehcah
 
Since you already have a 24 port hardware RAID card, you cheapest option would be hardware RAID 5/6, with whatever OS you are most comfortable with (linux with LVM/ext3, windows with NTFS).

ZFS solutions (nexenta, freeNAS, freeBSD) you would be better off selling the RAID card and getting JBOD controllers.

Storage for home media streaming doesn't need much performance, and "scalability" is normally less of a concern than expandability.
 
There's one problem in your plan; you bought a RAID controller and want to use ZFS. Those are bad combinations. You can create single-disk arrays or set the controller to "JBOD" or RAID-less mode, but my own Areca ARC-1220 does disconnect disks on timeouts even when set to JBOD; so this isn't a good setup for ZFS at all!

If possible, trade your Areca for a normal HBA like Intel SASUC8i or SuperMicro USAS-L8i. These are great for ZFS and should work under all OS (though disabling smartd and MSI interrupts may be necessary under linux).

Another option would be to use another system than ZFS, but without TLER-capable drives you would keep the problem of disconnecting disks. This will only be apparent when your disks encounter these, so if you got a full backup anyway this shouldn't be a big issue. But something tells me you don't want to backup this much space, right? So it's a bit of a dilemma i would say.

Think about this first; your hardware should match the requirements and optimal specs for the OS that you chosen. The rest of your hardware looks fine, though. I will add that 8GiB is on the low side for this many disks. 12 or 16GiB would be more logical; if money permits. 8GiB DDR3 ECC SDRAM goes for as low as 120 euro in my area; that's a steal i think and makes giving ZFS plenty of RAM quite affordable. The general idea is that you save money on RAID controller and BBU and TLER capable drives, and can spend those on more disks, more RAM and an SSD for cache.

So i'm not really sure what your best option is, given you already spent alot of money. There are good arguments for alot of solutions, but only you can tell which solution is best to you.
 
Thank you for the replies. I tried to find a cheap 8 Sata port card that I could purchase 3 of, and did not find any. Lot's of Sata cards available out of the US, but not from Canadian retailers I like to use.

How would Raid 6 performance compare to say Raidz under ZFS? I have 2 ubuntu servers running in my basement rack now, so sticking with a linux server doesn't bother me. I'm not sure where to look for how to implement my disk requirements. My gut is telling me to cut them up into either 4 groups of 6 or 6 groups of 4, rather than 2 groups of 12. Again, not sure what the performance impact of hardware Raid vs Software will be.

I had budgeted to lose between 4-8 disk to parity. My assumption all along was that I would see a maximum of 28-32TB usable out of 24 maxed out bays with 2TB drives.
 
Unless you are going past 1Gb ethernet, pretty much any 3 disk software or hardware RAID 5 will be enough to stream movies and music.

How you split them up depends on your space requirements. If you have 9TB of videos, making 6 groups of 4 drives would force you to break up that folder. Try to figure out what your largest single folder uses for space, and figure out how much it will grow in 6 or 12 months, then you will have a better idea for splitting up the arrays.
 
Ok. I've spent the past few hours since your last posts reading up on hardware vs. software RAID and trying to better understand what a Server OS like ubuntu might offer over NAS appliance's like I mentioned in my original post. Rather than confusing myself further along with the scenario I'm about to layout, can anyone point me toward some specific reading material on managing ubuntu server or home NAS.

We know I've purchased a 24bay Norco 4224 Chassis and Areca 1280ML 24 port SATA controller. I also purchased an additional 5 x 2TB drives to bring me to a total of 12 x 2TB. Because I was thinking about running a NAS appliance like install, I was going to use a 30GB SSD for caching and boot the OS off of a 8GB USB pen drive. In case it is pertinent, one of my 24 port switches is GB and all servers, htpc's and my office gear connect to it. The rest of my gear connects through 2 x 24port 10/100 switches.

The MOBO I am reusing is limited to 4GB and has a Core 2 Duo E4400. I did not plan to upgrade these components as I thought the Raid Controller would compensate for 2.5 year old hardware. Do I need to come up with another $500 to upgrade these?

So... Now that I have itemized what what I have. Can you help me figure out how I should piece all this stuff together using Ubuntu 10.04 LTS Server? My first thought is to install the OS on the SSD vs. mirrored 7200 RPM SATA drives. If this is not correct, can the SSD be used for additional caching or similarly to speed up performance of my system? I have a pile of smaller 7200RPM SATA drives that could be used for OS.

The last piece is to figure out how to mount my drives. For future expansion, I would like to upgrade by adding additional arrays in a quantity of drives (4, 6 or 8 based on maximum performance) that makes the most sense for RAID 6. I don't understand LVM vs RAID but would it allow me grow my storage pool in this way? I know I'm stating functionality that comes from FreeNAS or Nexenta. Just wondering if this can also be accomplished Ubuntu.

Any critiquing, finger pointing or if necessary laughing at me for my lack of understanding, is perfectly acceptable in this case. My head is full of somewhat useful information, but piecing it all together correctly is beyond me at this point.

e
 
Since you are limited currently to 4Gb of ram, have a hardware RAID card, and are still learning, something with a front end would be the best option. Either freeNAS without ZFS, or a linux equivalent (haven't looked at those in a while). This will make LVM/partitioning/mountpoints easier, letting you mount the array's and start sharing. I believe ZFS is the only consumer system that can use SSD's as cache, so you could use it to install the OS (or use a hardware RAID1 mirror for os).

You can use ubuntu or other general distro's, but you would need a much better understanding of LVM and growing filesystems before attempting that. Learning on your primary storage system is not a good idea. They don't have any sort of web frontend, so it would have to be all done from the command line.

Lastly, personally I would do a mirrored boot drive, and use all 12 drives you have in a RAID6 array (mount to /data, create whatever folders you need, /data/music, /data/videos, etc, and share). When you get more drives, you can make another RAID array, and split up your data between them, or if you are feeling crazy, use the new array to expand your LVM.

One that thing that always needs to be brought up. RAID is not a backup. Do not rely on anything you put together to save your data, layer your data.
 
Adi:

I'm not deliberately trying to abuse your patience, but I do have a few more questions. First off, I need to address the last statement of your last post. I very much agree that Raid doesn't replace a good backup. Building this server is project number 1. I may, if it makes sense, try to re-use another Tower Case and my external 8 bay enclosure for for some form of an 11 drive backup. The only flaw I see is that regardless of what I use, the backup will reside in the same house as my Rack. Other than data duplication, I'm not sure that offers enough protection to justify the cost of 11 more drives?

I'm going to try and get a bit more specific with my questions. Against yours and other peoples better judgement and suggestions, I do plan to try and use Ubuntu Server. Although I agree mixing some of the Functionality of FreeNAS might offer the GUI. Worst case scenario, I could look to Webmin (I know it is not supported for Ubuntu) as the GUI and make all modifications via command line, relying on Webmin for review only. On previous test box, I did install eBox, but hated it. Webmin may not be supported or recommended for Ubuntu, but at least it does not take over everything and become your only means of editing files. I hated eBox and had to do a complete rebuild after trying to remove it.

Back to my wacky scenarios...

If you had a total of 24 bays available, Hardware Raid for all 24 and were starting fresh with a minimum of 12 x 2TB drives and assume using Raid 6. How many array's would you want to break 24 bay's? I had initially planned on 2 groups of 12, but that would be extremely expensive to grow all at once next time. Not to mention that I think 12 Disk in one Raid6 Array have performance issues or could be riskier. If this isn't accurate than starting out with a 12 disk array and adding 2 or 3 smaller ones is still an option. My biggest fear for large arrays is the rebuild time if ever I want to cheat and swap out individual 2TB drives with larger ones when available to grow. As I understand it, this can be accomplished disk by disk after each is rebuilt. The parity drive(s) is last and once rebuilt the array magically appears larger based on new size of the individual disks in the array.

If it makes more sense, I am open to 3 groups of 8, 4 groups of 6 or even 6 groups of 4, which ever makes the most sense from a usable and expandability perspective. I had an additional $3K set aside for a new ripping workstation. I will gladly use some of its budget for more disk if necessary to bump from 12 to 16 or whatever. I should also point out that going into the new Norco 24 bay chassis, I still plan to reuse my previously mentioned MOBO with 4GB of RAM (mobo max) and a Core 2 Duo E4400 processor. Instead, should I be looking to upgrade this MOBO to even a slightly higher end Desktop MOBO, Processor and 16of RAM? Would it make enough difference?

Possibilities:
http://www.newegg.ca/Product/Product.aspx?Item=N82E16813182211
http://www.newegg.ca/Product/Product.aspx?Item=N82E16819115222
(2 or 4) http://www.newegg.ca/Product/Product.aspx?Item=N82E16820139077

Would these 3 upgrades provide any real, noticeable performance? Or would I have to spend an additional $500 for true server components?

My worst case scenario budgeted usable data amount was based on 4 groups of 6. In the case of using 2TB drives, I think that is roughly 16 x 2TB(1.8 actual) or 28.8TB available? I understand there is a formula to follow for best performance and minimizing the impact of lost disk. I'm not describing it correctly, but what I mean is... I remember reading somewhere that when spanning(quite possibly the wrong term?) or sharing multiple Raid 6 array's, you don't always lose 2 drives for parity and the actual amount of usable disk space can grow. There was also a few good examples of what array's should look like. I wish I could find that article as I know I've massacred what it intended to say.

How many array's would you create to start? Will volume manager allow me to add new array's later to grow my usable disk space without having to move folders around? There has to be some way of growing without a complete rubuild otherwise I would have to assume software Raid wouldn't exist. Part of my thinking comes from unRAID, which I currently use. As long as the parity drive remains the largest in the array, you can add any number, size of disk and grow 1 disk at a time. FreeNAS and Nexanta also also allow growth through adding vdevs(I think). A bit more complicated than unRAID, but still doable, I believe? I have given up on all the other solution and want to make this a 100% Ubuntu Server solution.

My first stop was chapters for reading material as well any other book site for good reference material. I have googled, trolled IRC and posted in may forums and can not find any real instruction or advice for what I want to do. Most have simply stated the obvious, I (me) don't have enough knowledge to go the Ubuntu route. Regardless, I'm going to attempt it. Catastrophes ahead be damned! I have been warned! I've got Ubuntu Server running on a few DL380's in my basement rack and have rebuilt them as part of my own personal education. What I haven't been able to learn there, VirtualBOx on my laptop has been great. My prod box which really does nothing (DHCP, DNS, Internet Sharing, DDNS for my home LAN) is now GUI less, not even Webmin. The fact that I celebrate that statement should horrify a few of you!

Any additional guidance and feedback is appreciated. I know this is a long a post and somewhat abusing the intents of this forum. If I've pushed to far, tell me, laugh at me, whatever. It's all good!

e
 
You must log in or register to reply here.
Back
Top