Building new storage server, need help

M

MatDef

Supreme [H]ardness
Joined
Nov 15, 2005
Messages
4,705
Like my title says, I'm currently putting together the parts I need for a new file server at the house and I'm a little stuck at the moment. I don't have a lot of room, and I've decided to make do with what I already have, at least for the case and mobo:

Mobo: SUPERMICRO MBD-X7SPA-HF-O
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182234

Case: Lian-Li Q08B
http://www.newegg.com/Product/Product.aspx?Item=N82E16811112265

For my last system I was using WHS with duplication enabled, but it was costing way too much to keep adding more storage as fast as I was using it, plus the lack of support for the larger 3tb drives and 4k sectors was extremely annoying too.

So my question is, what should I be using for my new setup? I loved the easy to use features of WHS, so something similar would be great (at least in terms of having a unified pool/single mount point for all my shares). I'm looking at buying at least 3x3tb drives off the bat to start, simply because with only 6xHDD bays my space is at a premium. I'm open to pretty much anything, both windows based and not (I'd like to keep everything software based, if it's not possible getting a hardware raid card would be an option). I'd also like to eventually be able to reuse the Sans Digital esata port multipliers that I currently have, so whatever OS I'm going to use should have support for them.

Any ideas? I'm not sure if I've been 100% clear on what I need, so feel free to ask me anything and I'll do my best to answer.
 
What kind of access do you have to Microsoft software (Technet, MSDN, etc)? Apparently you can run 2008 Hyper-V for free, and run WHS inside of that - but I'm not sure how that works with respect to the 3TB drives. I use Server 2008 R2, which is great if you don't have to pay full price, and I use FlexRAID with it (FlexRAID View, which I'm not using, gives you drive pooling as well).
 
Forceman said:
What kind of access do you have to Microsoft software (Technet, MSDN, etc)? Apparently you can run 2008 Hyper-V for free, and run WHS inside of that - but I'm not sure how that works with respect to the 3TB drives. I use Server 2008 R2, which is great if you don't have to pay full price, and I use FlexRAID with it (FlexRAID View, which I'm not using, gives you drive pooling as well).
Click to expand...

I'm trying to move away from WHS simply because while duplication is nice, it just costs too much (in terms of drive space and the cost of new drives). I was looking at FlexRAID View, but from what I've read it was still considered beta and not really ready for production use with important data (though correct me if I'm wrong).

I also have TechNet, so I can use pretty much any server OS if need be.
 
Not sure about FlexRAID View - I didn't need drive pooling so much, so I just went with straight FlexRAID, which has stable releases. FlexRAID View sounds intriguing though - I may switch in the future.
 
Here's something that I've working on for the past few days or so. Do note that the following is still very much a WIP and mostly covers solutions where you want to add different sized drives to a storage array of sorts:
Amahi with Greyhole:
Links:
http://wiki.amahi.org/index.php/Greyhole
http://code.google.com/p/greyhole/
http://www.amahi.org/

Pros:
+ Lets you use different size drives and combine them into one drive pool ala WHS's Drive Extender feature.
+ Can set the number of copies of a folder/file that you want stored on other drives. So unlike WHS where only a single copy of that data is stored on a different drive, you set it towher that file/folder can be copied across every single hard drive in your PC. So 5 drives means 5 copies of your data
+ Supposedly you're still able to remove a drive from that Greyhole server and still be able to read the files on another PC without greyhole installed.
+ Free and works with Linux, which is also free

Cons:
- Still very new and experimental. So not all of the bugs may have been fixed or discovered yet.
- GUI is still relatively clunky
- Takes extensive amount of time to recover and repool data across a new drive after a drive failure

Notes:
* Should you delete a file while using Greyhole, all the other copies are deleted as well.
* While the above example recommend Amahi Home Server, you can use Greyhole in virtually any Linux based OS

Flexraid Basic:
Links:
http://en.wikipedia.org/wiki/FlexRAID
http://www.openegg.org/FlexRAID.curi

Pros:
+ Uses data based parity in which parity data is only kept of the file actually stored, not like with traditional RAID where parity data is is kept of the entire hard drives, regardless of whether or not you have data on those drives.
+ Allows use of different sized drives
+ Free

Cons:
- Not realtime parity so if the data needs to be protected at all times, not a good idea to use FlexRAID
- The FlexRAID needs periods of time of no usage at all in order to re-sync the RAID
- If multiple users are editing stored files, flexRAID is not recommended

Notes:
* FlexRAID Live does not have the cons of the FlexRAID Base. However it has not been released yet and even it is, it would still be a bit experimental at that stage.
* A little old but still good reading:
http://www.overclock.net/htpc/694948-has-anyone-tried-flexraid-their-media.html

Unraid:
Links
http://www.lime-technology.com/
http://lime-technology.com/wiki/index.php?title=UnRAID_Wiki[/QUOTE]

Pros:
+ Allows use of different sized drives.
+ Should one or multiple drives die, only the data on those dead drives are lost.

Cons:
- unRAID Free has a limit of 3 drives (2 data, 1 parity).
- unRAID Plus has a limit of 6 drives (5 data, 1 parity, 1 cache)
- unRAID Pro has a limit of 21 drives (19 data, 1 parity, 1 cache)
- Costs $120 for unRAID Pro (21 drives and $69 for unRAID Plus (6 drives)
- Relatively low write speeds without that cache drive.

Notes:
* A little old but still good reading:
http://www.overclock.net/htpc/694948-has-anyone-tried-flexraid-their-media.html

ZFS File Server:
Links:
Building your own ZFS fileserver
FreeBSD ZFS NAS Web-GUI

Pros:
+ Free
+ Fairly resilient against crashes and such
+ Does small incremental backups of sorts

Cons:
- May be difficult to learn for a person who've never worked with BSD before
- Requires a large amount of RAM (4GB+ recommended)
- Cannot expand an existing RAID array with additional drives.

Note that some of the above is still not finished or completely checked out, at least it should give you an idea of what to look at .
 
Last edited:
MatDef said:
I'm looking at buying at least 3x3tb drives off the bat to start, simply because with only 6xHDD bays my space is at a premium.
Click to expand...

Buying 2TB drives will save enough money that you can buy more drive bays - either a different case or an external case.
 
Danny Bui said:
Here's something that I've working on for the past few days or so. Do note that the following is still very much a WIP and mostly covers solutions where you want to add different sized drives to a storage array of sorts:
Amahi with Greyhole:
Links:
http://wiki.amahi.org/index.php/Greyhole
http://code.google.com/p/greyhole/
http://www.amahi.org/

Pros:
+ Lets you use different size drives and combine them into one drive pool ala WHS's Drive Extender feature.
+ Can set the number of copies of a folder/file that you want stored on other drives. So unlike WHS where only a single copy of that data is stored on a different drive, you set it towher that file/folder can be copied across every single hard drive in your PC. So 5 drives means 5 copies of your data
+ Supposedly you're still able to remove a drive from that Greyhole server and still be able to read the files on another PC without greyhole installed.
+ Free and works with Linux, which is also free

Cons:
- Still very new and experimental. So not all of the bugs may have been fixed or discovered yet.
- GUI is still relatively clunky
- Takes extensive amount of time to recover and repool data across a new drive after a drive failure

Notes:
* Should you delete a file while using Greyhole, all the other copies are deleted as well.
* While the above example recommend Amahi Home Server, you can use Greyhole in virtually any Linux based OS

Flexraid Basic:
Links:
http://en.wikipedia.org/wiki/FlexRAID
http://www.openegg.org/FlexRAID.curi

Pros:
+ Uses data based parity in which parity data is only kept of the file actually stored, not like with traditional RAID where parity data is is kept of the entire hard drives, regardless of whether or not you have data on those drives.
+ Allows use of different sized drives
+ Free

Cons:
- Not realtime parity so if the data needs to be protected at all times, not a good idea to use FlexRAID
- The FlexRAID needs periods of time of no usage at all in order to re-sync the RAID
- If multiple users are editing stored files, flexRAID is not recommended

Notes:
* FlexRAID Live does not have the cons of the FlexRAID Base. However it has not been released yet and even it is, it would still be a bit experimental at that stage.
* A little old but still good reading:
http://www.overclock.net/htpc/694948-has-anyone-tried-flexraid-their-media.html

Unraid:
Links
http://www.lime-technology.com/
http://lime-technology.com/wiki/index.php?title=UnRAID_Wiki

Pros:
+ Allows use of different sized drives.
+ Should one or multiple drives die, only the data on those dead drives are lost.

Cons:
- unRAID Free has a limit of 3 drives (2 data, 1 parity).
- unRAID Plus has a limit of 6 drives (5 data, 1 parity, 1 cache)
- unRAID Pro has a limit of 21 drives (19 data, 1 parity, 1 cache)
- Costs $120 for unRAID Pro (21 drives and $69 for unRAID Plus (6 drives)
- Relatively low write speeds without that cache drive.

Notes:
* A little old but still good reading:
http://www.overclock.net/htpc/694948-has-anyone-tried-flexraid-their-media.html

ZFS File Server:
Links:
Building your own ZFS fileserver
FreeBSD ZFS NAS Web-GUI

Pros:
+ Free
+ Fairly resilient against crashes and such
+ Does small incremental backups of sorts

Cons:
- May be difficult to learn for a person who've never worked with BSD before
- Requires a large amount of RAM (4GB+ recommended)
- Cannot expand an existing RAID array with additional drives.

Note that some of the above is still not finished or completely checked out, at least it should give you an idea of what to look at .
Click to expand...

So it looks like both options that suit me best (greyhole and flexRAID view) are both still not quite ready for actual use. :(

ZFS looks good, but the fact that I can't gradually grow the array is a deal breaker for me at this point.

Thanks for all the info, it's been incredibly helpful.

GeorgeHR said:
Buying 2TB drives will save enough money that you can buy more drive bays - either a different case or an external case.
Click to expand...

Which would be good if I wanted to save money, but right now I want to get the most amount of storage in the smallest space possible. :p
 
MatDef said:
ZFS looks good, but the fact that I can't gradually grow the array is a deal breaker for me at this point.
Click to expand...

Just to be clear: ZFS uses a different method of storage expansion:
sub.mesa said:
What you cannot do, is expand an existing RAID-Z (RAID5) or RAID-Z2 (RAID6) array with one or more disks.
But, you can add new disks or RAIDs to an existing pool. So if you have a 4-disk RAID-Z, you can add another 4-disk RAID-Z so you have 8 disks. The second array would share free space with the first; in essence it would be a RAID0 of two RAID5 arrays. ZFS can expand this way.

What you can do, is expanding mirrors and RAID0's. In the example above that's what actually happened: a new array is RAID0-ed with the existing array. New created files will be written to both devices, for additional speed. Setting copies=2 would make files in that directory be stored on both RAID arrays; for extra redundancy.
Click to expand...

Still a deal breaker?
 
Danny Bui said:
Just to be clear: ZFS uses a different method of storage expansion:


Still a deal breaker?
Click to expand...

What OS would be best for ZFS usage?

That would actually work, I could pretty easily use 3x3tb RaidZ to start, then add another RaidZ array down the line when I needed more space. Or just buy 6x drives to start and make a RaidZ2 array.

Another question, right now I have two of these Sans Digital esata drive bays, could I reuse them? I have the crappy pcie esata card that came with both, but if I needed to replace it to get one that had better driver support I would.
http://www.newegg.com/Product/Product.aspx?Item=N82E16816111080

They weren't exactly cheap and if I could attach them down the line when needed for another raidZ array, it sounds like it would pretty much do what I want.
 
MatDef said:
What OS would be best for ZFS usage?
Click to expand...
FreeNAS for simplicity
FreeBSD for large setups.
MatDef said:
Another question, right now I have two of these Sans Digital esata drive bays, could I reuse them? I have the crappy pcie esata card that came with both, but if I needed to replace it to get one that had better driver support I would.
http://www.newegg.com/Product/Product.aspx?Item=N82E16816111080

They weren't exactly cheap and if I could attach them down the line when needed for another raidZ array, it sounds like it would pretty much do what I want.
Click to expand...
Dunno to be honest. Hopefully our resident ZFS guru (sub.mesa) will be along and help you out here
 
MatDef said:
What OS would be best for ZFS usage?
Click to expand...
For systems with 4GiB RAM and below Solaris-derived OS will be faster, FreeBSD remains a safe choice for the long term and has distinct advantages, just like Solaris has over FreeBSD.

FreeNAS is actually FreeNAS 0.7 and FreeNAS 0.8, which is a complete rewrite by different people, essentially a new project baring the same name. The differences are huge. As for development status, FreeNAS 0.8 could be roughly compared to my project, in the sense that it lacks certain core features, but should be 'usable' given some rough edges. Still the most stable solution right now would be a normal FreeBSD build or FreeNAS 0.7. Solaris based derivatives are also very nice, try a few before you make up your mind.

Keep in mind that if you keep your pool at Zpool version 15 and ZFS filesystem version 4, that you are able to switch between any Solaris based ZFS system and FreeBSD 8.2 and up.

Another question, right now I have two of these Sans Digital esata drive bays, could I reuse them? I have the crappy pcie esata card that came with both, but if I needed to replace it to get one that had better driver support I would.
http://www.newegg.com/Product/Product.aspx?Item=N82E16816111080

They weren't exactly cheap and if I could attach them down the line when needed for another raidZ array, it sounds like it would pretty much do what I want.
Click to expand...
These are Silicon Image port multipliers; not exactly high-performance but they should work in FreeBSD. The siis-driver supports Silicon Image controllers quite well, including hot-swap, NCQ, port multipliers and even TRIM on SSDs. But the hardware bottlenecks (one SATA/3Gbps going to 4 HDDs) will remain, of course.

You could use my latest livecd to test whether they work; they should pop up under the disks page, probably as separate disks (4). Note that if you created a 'fakeRAID' array, that FreeBSD will detect this and use pseudoraid driver to apply its own software RAID to match the RAID structure as provided by the Silicon Image metadata, which is stored in each drive.

You should use GPT formatting on your disks, leaving at least 1MiB as unused (the default). This ensures the metadata will not be touched. GEOM formatting may give you problems! This also means that you cannot migrate to Solaris; since it does not support the freebsd-zfs GPT partitions, apparently.
 
First of all, thanks to both Danny_Bui and sub.mesa for your help so far, I've been busy the last two weeks playing with the server. I went ahead and went with FreeNAS 0.7 with zfs and have been very impressed with the performance/features so far. Right now I have a pool with a single 3 drive raidz array. My question is, how do I add another 3 drive array to the same pool without being forced to destroy it and create it again from scratch? I apologize if this is a stupid question, I'm still learning both ZFS and FreeNAS. :p

I'll provide screenshots of my setup if that's required, just let me know.

Edit: While looking at new drives, I noticed newegg is selling the Hitachi 7200rpm 3tb model for significantly cheaper than amazon is for the WD 3tb GP models. Would buying 3 of the hitachi's cause a problem with the current wd drives I have if I put them in the same pool?
 
Last edited:
I'm not sure how to do it in FreeNAS, I used and currently using OpenSolaris, I'm guessing its some sort of WebGUI front end, but look where you have an ability to manage devices/storage pools.

When you expand a zpool, all you're doing is adding an addition vdev to the pool. You currently have one vdev in a 3 drive raidz array. The next vdev can be of any configuration, even a single drive, but the rule of thumb is, don't screw yourself by lowering the redundancy (4 drive raidz vdev would also be lowering the redundancy).

This is where initial planning could help, but don't be afraid if you have to migrate to a new pool, its not that hard to set up.

This is how I know how to do it:
http://download.oracle.com/docs/cd/E19120-01/open.solaris/817-2271/gazgw/index.html

Personally I went from 1x4-drive raidz1 to 2x4-Drive raidz1 using vdev expansion, and this was reusing an existing raid6 array's with "raid" SATA drives. Then I bought 8 more regular drives (when I needed the space) put it in new storage pool (raidz2) and moved (zfs send/receive) the existing pool to the new one. Brought down the old pool, had to make some changes to CIFS and changed from the old iSCSI way to Comstar (since Comstar was the only supported method in new builds), then added the old drives back as another 8xdrive raidz2 vdev.

ZFS has been a pleasure to use at home, and this is someone that has never more than cursory touched any Unix/Linux/BSD based OS. I can't imagine using anything else in a home environment that needs large storage requirements, that's scalable, and also has paranoid level reaching data integrity and redundancy requirements. Oh and sweet sweet performance is nice. Really happy with my performance over GigE.

PS I also have 3 Hotspares... hahaha... I'll probably trim that down once I get an expansion case.
 
Because you mentioned 3 TB drives, those available are 5 platter models as far as I know. I cannot recommend these for a higher density file server, as these are loud, hot and cause a lot of vibrations. Even my 4 platter drives are running hotter in idle than the 2 platter drives with non-stop seeking. The 3 platter 2 TB drives (Samsung F4EG) are the best choice at the moment.

EDIT: It seems that the 3 TB WD drive only has 4 platters, but Seagate and Hitachi use 5 platters for their 3 TB drives. I still think 3 platter drives are better for fileservers.
 
Last edited:
The WD 3TB is 750GB platters; but low performance, lower than the 2TB Samsung F4. The cost per gigabyte of the 3TB drives is about twice as high, which is extreme i think. Using 2TB drives means that performance per drive will be faster, more drives means more performance as well, more capacity per dollar invested and also a lower redundancy overhead since having more disks using the same scheme would 'waste' less space.

Especially the Hitachi 3TB 7200rpm with 7W+ idle power consumption would be a monster that needs constant cooling or at least some airflow. I personally wouldn't buy these; the Samsung F4 2TB is very cheap right now and you can buy almost 3 samsung's (=6TB) for one 3TB drive.

A NAS server generally starts with a good controller or HBA; consider getting a 1068E so you can power up 14 disks (8 on the controller + 6 on your motherboard).
 
sub.mesa said:
The WD 3TB is 750GB platters; but low performance, lower than the 2TB Samsung F4. The cost per gigabyte of the 3TB drives is about twice as high, which is extreme i think. Using 2TB drives means that performance per drive will be faster, more drives means more performance as well, more capacity per dollar invested and also a lower redundancy overhead since having more disks using the same scheme would 'waste' less space.

Especially the Hitachi 3TB 7200rpm with 7W+ idle power consumption would be a monster that needs constant cooling or at least some airflow. I personally wouldn't buy these; the Samsung F4 2TB is very cheap right now and you can buy almost 3 samsung's (=6TB) for one 3TB drive.

A NAS server generally starts with a good controller or HBA; consider getting a 1068E so you can power up 14 disks (8 on the controller + 6 on your motherboard).
Click to expand...

While I do appreciate everyone trying to save me money, the #1 main concern in this build was stuffing as much storage as possible the chassis I have, which only has room for 6x3.5" drives (7 if I don't use an optical drive). I am aware how much cheaper the 2tb drives are, but at this point the extra cost is well worth the ability to fit more storage into the smaller space, and I already have 8x2tb drives as it is. :p

Thanks for the info about the hitachi drives, I guess the more expensive WD's are still the better deal after all.

I haven't copied much of my data over to the new server, so I'll play around with adding the new array to the same pool, if for some reason I fuck it up I still have the data elsewhere. :eek:
 
You must log in or register to reply here.
Back
Top