New FileServer - Advice and Comments welcome

J

Johnny-Gear

n00b
Joined
May 22, 2011
Messages
7
Hi HardForum,

I have enjoyed browsing these forums for some time, and thought my impending file server build to be a good chance to contribute.

This file server is mainly for home use, however, I work in IT, and I like to over design things as I feel spending slightly more in most cases net's better results/return on investment.

My main focuses here are reliability, efficiency(cost and power) and performance. The OS will either be Ubuntu Server 10.04 or 11.04 as I am most comfortable with that Linux distro and there is very little it cannot do. Below I have made a list of hardware components(and would love any/all comments on them):

1x SRCSATAWB (Intel RAID Controller - http://www.intel.com/products/server/raid-controllers/srcsatawb/srcsatawb-overview.htm)

1x MBD-X7SPA-H-O (SuperMicro Atom Board - http://www.supermicro.com/products/motherboard/ATOM/ICH9/X7SPA.cfm?typ=H)

1x KVR667D2S5/2G (Kingston DDR2 RAM - http://www.ec.kingston.com/ecom/con...o.asp?root=&LinkBack=&ktcpartno=KVR667D2S5/2G)

1x FD-CA-ARRAYR2-BL (Fractal Design Case w/ PSU - http://www.fractal-design.com/?view=product&prod=43)

6x HDS723020BLA642 (Hitachi 2TB HDD - http://www.hitachigst.com/internal-drives/desktop/deskstar/deskstar-7k3000)

1x HTS723216A7A364 (Hitachi 160GB 2.5" HDD - http://www.hitachigst.com/internal-drives/mobile/travelstar/travelstar-z7k320)

I was thinking I would use RAID Level 6 (5 active / 1 hot spare) but was also considering RAID 5. Comments and advice on this would also be appreciated.

Thanks for reading guys!

Johnny Gear
 
Raid6 is not 5 active and 1 hot spare. It is 4 data and 2 parity. If the data is important, raid6 is a good idea, since raid5 leaves you a window where the array is being rebuilt, such that another drive failing loses you everything. With 12TB of potential data, a rebuild could take days...
 
I wont comment on server and main board hardware. The sky is the limit in proportion to your pocket book.

However as far as raid is concerned. Never would I personally recommend a R5 with that much data. I would consider a raid 6 and honestly given the options that is about it. You can have the fastest most fantastic highest output 20 SSD raid 0 array but you will always be as fast as your layer 1 and 2 devices. Your NIC and your LAN.

Unless you plan on bonding multiple NICs with an 802.3ad capable switch you are only going to operate at 1gb/s.

In keeping with your question once again I recommend a RAID 6 array. You can sustain 2 disk failures and still be able to rebuild your array. You will NEED a hardware raid controller to utilize raid 6 as it would absolutely kill a fake raid solution like Intel ICH anything or AMD anything.

Raid 6 doesn't actually store parity on 2 drives, it stores parity on all 6 drives using 2 stripes of P, unlike 5 which only stores 1 stripe of ,P while using some super crazy math algorithm I was never paid to figure out. The way the parity is spread is important on how it can handle 2 drive losses at once and still work.

Also get 7 drives. If you experience a failure take one off of the shelf and slap it in place of the bad one. Hot spares are designed with High Availability database servers in mind where it is absolutely critical that the server NEVER leave production. Its another moving part you do not need to worry about.

I am not sure of the cost of the card you have listed there however I also recommend you consider:

Adaptec 6405 or LSI Mega-anything. They are lower cost R.O.C. devices with a fair amount of memory. You see people harping on needing 1-2GB on a raid controller, well those people do not think before they open their pie holes or have never had experience with raid arrays. 256mb for what you are building will be more than plenty. I run my 5805 adaptec with 512mb with 8 HD204UI (Desktop drives 24/7 with no issues thank God). You want a raid card with 1 -2GB of RAM on it fine go pay the premium and while you are at it go ahead and get the 10gb LAN hardware to go with it.

Battery backups are optional on the raid card. I say get a battery backup for your raid card if you are not planning on running a UPS or have the extra cash to get one. If not you will be surprised how USELESS raid write caching is if your primary point of access is a single 1gb NIC.

I am sure all the Navy Seals of RAID technology on these forums are going to go weapons hot on me for saying that but I ask them to prove me or the idea wrong.
 
Last edited:
tangoseal said:
I wont comment on server and main board hardware. The sky is the limit in proportion to your pocket book.

However as far as raid is concerned. Never would I personally recommend a R5 with that much data. I would consider a raid 6 and honestly given the options that is about it. You can have the fastest most fantastic highest output 20 SSD raid 0 array but you will always be as fast as your layer 1 and 2 devices. Your NIC and your LAN.

Unless you plan on bonding multiple NICs with an 802.3ad capable switch you are only going to operate at 1gb/s.

In keeping with your question once again I recommend a RAID 6 array. You can sustain 2 disk failures and still be able to rebuild your array. You will NEED a hardware raid controller to utilize raid 6 as it would absolutely kill a fake raid solution like Intel ICH anything or AMD anything.

Raid 6 doesn't actually store parity on 2 drives, it stores parity on all 6 drives using 2 stripes of P, unlike 5 which only stores 1 stripe of ,P while using some super crazy math algorithm I was never paid to figure out. The way the parity is spread is important on how it can handle 2 drive losses at once and still work.

Also get 7 drives. If you experience a failure take one off of the shelf and slap it in place of the bad one. Hot spares are designed with High Availability database servers in mind where it is absolutely critical that the server NEVER leave production. Its another moving part you do not need to worry about.

I am not sure of the cost of the card you have listed there however I also recommend you consider:

Adaptec 6405 or LSI Mega-anything. They are lower cost R.O.C. devices with a fair amount of memory. You see people harping on needing 1-2GB on a raid controller, well those people do not think before they open their pie holes or have never had experience with raid arrays. 256mb for what you are building will be more than plenty. I run my 5805 adaptec with 512mb with 8 HD204UI (Desktop drives 24/7 with no issues thank God). You want a raid card with 1 -2GB of RAM on it fine go pay the premium and while you are at it go ahead and get the 10gb LAN hardware to go with it.

Battery backups are optional on the raid card. I say get a battery backup for your raid card if you are not planning on running a UPS or have the extra cash to get one. If not you will be surprised how USELESS raid write caching is if your primary point of access is a single 1gb NIC.

I am sure all the Navy Seals of RAID technology on these forums are going to go weapons hot on me for saying that but I ask them to prove me or the idea wrong.
Click to expand...

Thanks for the in depth reply.

The controller I listed is an LSI ROC controller re-badged for Intel. The reason I prefer this one is the LSI chip without the LSI price tag(it is still not a cheap controller though).

Thanks for the advice on the drive configuration. I was going to go with a mounted(hard) hot spare, but I think maybe have 6 active drives and one on the shelf might actually be a better option.

Keep those comments coming guys :)
 
Are you short on space? I wonder why you went with the mini-ITX size. Have you considered micro-ATX with a Westmere CPU? Then you could drop the RAID card and just use mdadm for software RAID.

With Supermicro motherboards, especially for servers, it is usually best to get a -F model so that you have remote control via IPMI.

For the HDDs, why choose 7200 rpm? For a lower power file server, you'd be better off with 5400 rpm drives.

Note that HDDs that have been sitting on the shelf for several years may fail to spin up. If you are keeping HDD(s) on a shelf, you should probably power them up at least once a year.
 
Last edited:
john4200 said:
Are you short on space? I wonder why you went with the mini-ITX size. Have you considered micro-ATX with a Westmere CPU? Then you could drop the RAID card and just use mdadm for software RAID.

With Supermicro motherboards, especially for servers, it is usually best to get a -F model so that you have remote control via IPMI.

For the HDDs, why choose 7200 rpm? For a lower power file server, you'd be better off with 5400 rpm drives.

Note that HDDs that have been sitting on the shelf for several years may fail to spin up. If you are keeping HDD(s) on a shelf, you should probably power them up at least once a year.
Click to expand...

Micro-ATX is moving towards a Mid-Tower size which is far to big for a simple file server especially when I can fit 6x 3.5 and 1x 2.5 drives in my current case choice.

Also, a Westmere CPU for a file-server? Seems very over kill to me.... unless of course I were to run software RAID which is easily outperformed by a hardware raid controller anyway. Software raid implementations also have far less features generally. This is just my knowledge mixed with some informed opinions so if you can convince my otherwise I would be interested to hear your response.

I run a similar SuperMicro board for my router/firewall/transparent proxy. It has the IPMI feature and I found that I did not really get a whole lot of use out of it. If I were planning on having this somewhere other than home it may be important to me, but it is not.

The controller I am considering does not have a huge cache so I thought 7200 rpm drives would perform far better than 5400's. This opinion is based on educated guess work, so if you can convince me otherwise with some factual info I would appreciate.

Thanks for the info on non powered HDD's; Didn't know that could happen. Powering it up every so often wouldn't be too hard with a drive bay.

Johnny
 
Johnny-Gear said:
Also, a Westmere CPU for a file-server? Seems very over kill to me.... unless of course I were to run software RAID which is easily outperformed by a hardware raid controller anyway. Software raid implementations also have far less features generally.
Click to expand...

Apparently you draw a fine line between "overdesign" and "over kill". Too fine for me, anyway.

I thought you were familiar with linux. You should probably read up on mdadm, since you could not be more wrong about linux software RAID.
 
Well there is a trade off Johnny.

If you go with the 5400 RPM drives you probably wont find reliable enterprise drives. However they are going to work fine on a good hardware controller.

If you go 7200rpm, sure they burn a few watts more under load, but you will be able to utilize enterprise SATA drives that are rated for 24/7 use. Just two considerations to be made.

Also consider desktop drives on a soft or fake raid controller will or can fail due to error recovery timeouts that the disk probably do not have such as West Digital TLER. Hardware based controllers usually dont care about it that much since they are far superior in handling disk errors.

Lastly if you want absolutely durability, warranty, and 24/7 for millions+ of hours there are true SAS drives however be prepared to pay out the yazoo for them. The Seagate 15K cheetah sas drives @ 300GB are roughly $250/disk give or take.
 
john4200 said:
Apparently you draw a fine line between "overdesign" and "over kill". Too fine for me, anyway.

I thought you were familiar with linux. You should probably read up on mdadm, since you could not be more wrong about linux software RAID.
Click to expand...

Well, you mention lower power HDD's in the same breath as a major CPU upgrade. You are being mildly contrary. I would consider an Atom CPU, with the provision of a hardware raid controller, to be a good distributed load setup with the benefit of a long list of on card RAID features. How does your suggested upgrade improve on my current design? Please elaborate rather than making uninformative and snide comments.

Perhaps I do need to read up on mdadm, as I am only familar with RAID 1 setups using it. In general I prefer a hardware controller for previously stated reasons. Does mdadm handle RAID better than a hardware controller?

Johnny
 
john4200 said:
I thought you were familiar with linux. You should probably read up on mdadm, since you could not be more wrong about linux software RAID.
Click to expand...

+1

john4200 is right about this. Actually software RAID probably has the most features of all the types of RAID and is not limited by the controllers firmware, which, unless a very high-end and decent controller is purchased, the firmware may be very lacking in features.

Another thing, if you decide to use encryption at all, you can never have enough CPU speed and/or cores. My 5000+ AM2 CPU is struggling to keep up with my RAID5 arrays with encryption. I could only imagine using RAID6 with encryption and continue getting decent performance without a modern tri/quad-core CPU or better.

If you don't go with encryption though, RAID6 still has a lot of calculations and with software RAID and a multi-use environment, having a fast CPU is key to performance.
 
The reason why I was and am leaning towards hardware RAID is because it is a simple fact that a piece of code can do something and a hardware implementation of that same 'code' can do it quicker and in a lot of cases better. (edit: yes I do understand there is a firmware which is 'software' but I feel this is still a valid statement)

I understand as with many Linux utilities mdadm is highly configurable and potentially a good solution.

My question to everyone recommending software RAID, over hardware, is where do you draw the line? How do you compare system specs to a hardware RAID controller?

Does anyone have any experience with the differences(performance wise) between software and hardware RAID 6(or 5)? (and what was the controller / specs)
 
Last edited:
The difference is primarily cost and your reasons for needing it.

I like software RAID and use it for the facts that it is feature-rich, compatible across all Linux distros that use MDADM and Debian, and is high-performance for a multi-user environment.

However, if I were running a data-center or an enterprise-class server in a many-user environment, I would go with hardware RAID, hands down. Having that amount of load taken off of the CPU is a must, and a BBU and additional cache is very necessary when the HDDs are being utilized.

imo, for home use, hardware RAID is a waste of money unless you are doing some serious number crunching on the array(s) such as 3D imaging and rendering, or HD video encoding.
 
Red Falcon said:
The difference is primarily cost and your reasons for needing it.

I like software RAID and use it for the facts that it is feature-rich, compatible across all Linux distros that use MDADM and Debian, and is high-performance for a multi-user environment.

However, if I were running a data-center or an enterprise-class server in a many-user environment, I would go with hardware RAID, hands down. Having that amount of load taken off of the CPU is a must, and a BBU and additional cache is very necessary when the HDDs are being utilized.

imo, for home use, hardware RAID is a waste of money unless you are doing some serious number crunching on the array(s) such as 3D imaging and rendering, or HD video encoding.
Click to expand...

Thanks for the explanation.

What sort of system specs (CPU/RAM/Board/Drives) would you recommend for good performance(limited by gig network) on a RAID 6 array?

Thanks again for all the advice.

Johnny
 
So I have looked into the hardware changes required if I were to do away with the hardware RAID controller in favor of a software RAID and I would still need some sort of SATA HBA as no mini-ITX boards have any more than 4x SATA ports as far as I can tell.

I found this HBA which looks like it would do a great job at handling all my RAID devices:

SuperMicro - AOC-USAS2-L8e ( http://www.supermicro.com/products/accessories/addon/AOC-USAS2-L8i.cfm?TYP=E )

Just need to consider what a good choice for board / CPU would be if in fact I decided on a software RAID array instead. Not sure whether I should be looking for more cores or straight power?

Johnny

edit: just found an informative and concise brief on pro's for each side which is making software RAID look more favorable. ( http://linux.yyz.us/why-software-raid.html )
 
Last edited:
thefreeaccount said:
The sad truth of this thread:
Johnny-Gear is your boss.
Click to expand...

LOL, even down to the politely-worded demands for more information to convince him!

For this type of server, I think the Supermicro X8SIL-F is the best motherboard choice. It has an onboard USB plug so the OS can be booted from USB, leaving the 6 onboard SATA ports free for HDDs for software RAID. Paired with a Westmere CPU, the idle power would be quite low. Not as low as an Atom, but probably lower than an Atom and a RAID card.
 
You must log in or register to reply here.
Back
Top