• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

storage server build questions

S

schnell

Gawd
2FA
Joined
Jul 22, 2005
Messages
764
I work for a broadcasting/production house and we are looking to consolidate our storage. Currently we are using 4 different NAS systems each with several terabytes of storage. We don't have the money to buy a SAN or any kind of off the shelf storage system, so I have been asked about the feasibility of building a system for around 5k. This will mainly be a storage tank for video and project files as well as some personal document storage. I will most likely also setup a partition for me to do some backup storage from our other servers.

I am looking into the Supermicro cases instead of the Norco 4224 because after adding redundant power the extra cost is minimal plus with the Supermicro case I can get a built in SAS expander.

Parts so far:
Supermicro SC846E1-R900B
Supermicro MBD-X8SIL-F-O
Intel Xeon X3440
KVR1333D3E9S/4G x2
HITACHI Deskstar 7K3000 x24
2 500gig drives in RAID1 for the OS


I am taking suggestions on RAID cards. I need something that will do raid 6, support SAS expanders, support multiple raid arrays, support 2tb drives, and won't break the bank.

As far as the OS is concerned I have a copy of server 2003 or 2008R2 that I can use or I am also looking into the possibility of openfiler.

I know all the ZFS guys are going to tell me to use zfsguru or the like and I have been researching that but I would rather stay with a hardware RAID solution as I am much more comfortable with that and it will be easier to support by people other than me (I don't plan on working here forever).
 
schnell,

I'm a hardware raid guy, at least used to be. I'm recently researching into ZFS, which really does look very good over a raid 5/6. I'm currently running an adaptec 5805. It's overall a good controller, but has some strange behaviors with hard drives not on the compatibility list. People are saying good things about Areca controllers, which also has good compatibility with HP SAS expander. Not sure what's your budget.

Chris
 
My advice:

Check the Supermicro HCL for options that support your expander chassis. Supermicro apparently does some tweaks to the expanders so it makes sense to go with something on the HCL if you are spending this much.
 
czhao - my main issue with that is I am the only one here who knows linux/unix and I would prefer not to leave them with a system they can not support in the future. I am interested in ZFS but I do not think my knowledge on the subject is high enough to use it on a production machine. If it were for my own personal use I would be all about trying something new but I can not afford to have the data non-accessible for a day to 2 while I figure it out at work.

oldschool - I do not need a SAS expander it is built into the chassis.

pjkenned - I did not realized they had an HCL list for raid adapters. Looks like the Supermicro AOC-USAS-H4iR will fit the bill pretty well.
 
I would recommend picking up one of the newer models like the SC846E16-R1200B instead of the one you listed. As previously suggested, the Areca cards work quite well and the 1880i would fit your needs.
 
While I'm a fan of Supermicro cases in general, I wouldn't personally buy a Supermicro chassis with an integrated expander, problem is Supermicro seem to tweak the firmware on it and there are idiosyncrasies often reported. I agree that if you need dual PSU's then the cost difference from a Norco isn't that major. IF you really want a Supermicro with an integrated expander then make sure to get whatever model has the newest generation LSISAS2x36 chip on it, because that's the latest/greatest expander chip on the market.

My friend ordered a couple SM cases a few weeks ago with "integrated expanders" and they ended up being the older SAS-1 LSI expander chips, so his drives only negotiate at 3.0Gbps and he can only use a single link to the backplane.

Unless its a pure cost issue then I think its still the most ideal to keep the backplanes dumb and drive them with card-based expanders and host adapters, that way everything is more interchangeable and modular, and can be upgraded as technology changes.

SIDENOTE: Also consider the Supermicro X8SIA-F, as it has a lot more PCIe slots.
 
Last edited:
odditory said:
Unless its a pure cost issue then I think its still the most ideal to keep the backplanes dumb and drive them with card-based expanders and host adapters, that way everything is more interchangeable and modular, and can be upgraded as technology changes.
Click to expand...

Completely agree. I have had a number of compatibility problems with the expanders in Supermicro backplanes.

Safest thing would be to avoid an expander altogether if you can. Choose a RAID card with 24-ports like an Areca 1880, and choose HDDs on Areca's HCL.
 
@OP

A couple of points.

a) You can get away with 250GB hard drives for the OS. Make sure they are enterprise class drives though.

b) If you are considering any kind of software RAID then yes, a Xeon and the associated ECC memory is a good idea. For hardware RAID, I'm not sure that a Xeon CPU will be of much benefit, since the controller has ECC memory on board anyway.

Can others chime in on this?

c) ZFS is a good choice if you value your data. Granted, it hasn't been around forever and we all had to use what was available until it came along, but since it's here, I'd recommend using it. If you go the Linux route, use JFS or XFS for the filesystem. ext4 is solid, but the tools to look after it can't see past 16TiB.

One advantage of using Linux (with LVM) or FreeBSD with ZFS is volume management. You can dynamically vary the space you dedicate to this data or that data. I'm not sure Server 2008 supports this. Can anyone offer info on this?

d) For hard drives I'd recommend Western Digital RE4-GPs, 2TB units. The RE4-GPs are low power (important when stringing lots of these drives together) and come with a 5 year warranty.
 
parityboy said:
@OP
a) You can get away with 250GB hard drives for the OS. Make sure they are enterprise class drives though.
b) If you are considering any kind of software RAID then yes, a Xeon and the associated ECC memory is a good idea. For hardware RAID, I'm not sure that a Xeon CPU will be of much benefit, since the controller has ECC memory on board anyway.
c) ZFS is a good choice if you value your data. Granted, it hasn't been around forever and we all had to use what was available until it came along, but since it's here, I'd recommend using it. If you go the Linux route, use JFS or XFS for the filesystem. ext4 is solid, but the tools to look after it can't see past 16TiB.
One advantage of using Linux (with LVM) or FreeBSD with ZFS is volume management. You can dynamically vary the space you dedicate to this data or that data. I'm not sure Server 2008 supports this. Can anyone offer info on this?
d) For hard drives I'd recommend Western Digital RE4-GPs, 2TB units. The RE4-GPs are low power (important when stringing lots of these drives together) and come with a 5 year warranty.
Click to expand...

a)I am using 500's for extra space for some apps that will be running on the server.
b)If I go with hardware RAID I guess I could look into a c2q but XENON just seems like the right thing to do in a server regardless.
c)As I said before I am the only one here who knows linux/unix at all. I don't want to leave them with a system they can not support if I change jobs. I also can not afford to have the system down while I research a problem with the OS.
d)Those drives are well out of the budget. 2tb re4 is over the budget with just 24 of the drives alone!
 
The X3440 is a decent choice and going with anything 771/775 at this point would be a poor choice. I would also skip the UIO cards like the USAS-H4iR since you don't have UIO slots on the board (though it will technically still work).
 
Blue Fox said:
The X3440 is a decent choice and going with anything 771/775 at this point would be a poor choice. I would also skip the UIO cards like the USAS-H4iR since you don't have UIO slots on the board (though it will technically still work).
Click to expand...


Good points.

As for the motherboard, I would tend to agree that an ATX board may be better for more expandability.

If you are going to have 48TB and ZFS, I would offer that you may want 12-16GB of RAM if possible. 8GB is really a bottom limit. I think most of the ATX Supermicro motherboards let you use six slots with RDIMMs and four with UDIMMs.
 
schnell said:
d)Those drives are well out of the budget. 2tb re4 is over the budget with just 24 of the drives alone!
Click to expand...

I know enterprise-class drives are expensive, but they are also higher-binned than the consumer level stuff, and should be less likely to suffer from surface defects and other things that can corrupt your data.

If you go for enterprise-class drives, obviously you'll pay the extra, but (in combination with a good hardware RAID card) it takes away some of the headache of achieving and maintaining data integrity - 48TB is a lot of data to corrupt.

If you go the consumer drive route, you'll save money, but you'll increase the risk of data and/or filesystem corruption, which is why myself and others have put forward ZFS as the best option. Additionally, you may wish to look at the 5K3000 models rather than the 7K3000 - since the server will be accessed over GigE, I doubt 7200RPM drives will make a difference, and with that many drives, you'll save more than a few Watts.
 
parityboy is spot on. The Hitachi 5K3000 benchmarks actually surprised me as I was expecting a slower drive. I have four at this point and they all benchmark similarly so I feel fairly confident in their speed. For video I/O which is mostly sequential reads/ writes, in RAID 6 over 1-2 GigE links the performance difference is most likely not a huge concern but there is a good chance the annual bill will be $25-100 less with the 5K3000 over the 7K2000 and that many drives.
 
Good luck, curious to see your outcome

But I have been stuck in the hurt, no zfs :( Woe is me
 
We have a similar need at work. The basic requirement is to store 40TB of contiguous data as cheaply as possible.

After comparing the time/effort I would need to build my own w/a Norco 4224 and supermicro motherboard and sas card/expander. I broke down the project into 2 parts. The storage case and the server.
For the server, I'm looking at the Dell R310 and pairing it with the Dell SAS 6 HBA or the Dell PERC H800. The difference is $600 for the raid card. The Dell R310 supports 2 x pcie 8x cards so we can use one for SAS controller and one for dual/quad GBe nic or another SAS controller.

I'm thinking of going with the NORCO DS-24E for the storage part and using the Seagate Barracuda drives for the 5 year warranty and past performance. Although, I am looking at the Hitachi drives...

Dell R310 $1200-$3000
NORCO DS-24E $1400
24 x 2TB Sge 7k drives @ $170 = $4100
or
24 x 2TB Sge 5k drives @ $100 = $2400
external SAS Cable <$100

So we can go as cheap as $1200 + $1400 + $2500 = $5000 for non raid slower drives
or $3000 + $1400 + $4200 = $8600 for raid and faster drives. The business need will determine whether we go with the better barracuda drives or just settle with the slow drives. We're also planning on keeping a couple cold spares. The real cost variable comes down to the server and with this method, we can buy cheap server now, and just swap out the server later.
We can expand this storage by adding another DS-24E + drives for $4000-$5600 or utilize 3TB drives next year.

We also looked at picking up several 2U QNAP 859+ for $2000 plus cost of drives and using ISCSI to chain them together. But that was more work and still seemed like a hack to get 40TB of contiguous space. It was $6000 to house 24 drives versus $2500-$3500 to house the same 24 drives.

I might pick up a Dell 6Gbps HBA to test with the HP SAS expander... If I really want an out of the box solution I might get the DS-24E to test with...

Let us know what you do and post a build log if you decide to build your own :-D
 
If you want a tested solution:
SMC X7DWN+ in 846E1 chassis with AOC-USASLP-L8i and ST32000542AS disks
I have had about 150Tb in three of these for over 12 mths.
Easy to expand with a 846 JBOD.
Performance is reasonable for the cost, and they don't exhibit the usual timeout issues on some disks with expanders.
I'm testing the Hitachi disks soon.
 
WiseOldOne said:
If you want a tested solution:
SMC X7DWN+ in 846E1 chassis with AOC-USASLP-L8i and ST32000542AS disks
I have had about 150Tb in three of these for over 12 mths.
Easy to expand with a 846 JBOD.
Performance is reasonable for the cost, and they don't exhibit the usual timeout issues on some disks with expanders.
I'm testing the Hitachi disks soon.
Click to expand...

What OS are you running? Raid level? Have you had any drive failures in 12mo?
 
Dear OP,

1. For broadcasting/production house facility where you depend on it for everything, perhaps you need more than that.

2. Very 1st thing is rough non-committed project budget so that we can assess various options.

3. And I explain here because budget-wise if you are floating between in gray area, you can have one full-scale high-octane setup, or two lower-cost lower-scale servers that protect each other, or hybrid that complement each other

4. For example, a primary facility with hardware-RAID and certain amount of enterprise disks, then a secondary much cheaper facility that employs non-RAID controllers and cheaper/slower/larger-capacity drives

5. And you probably need redundancy setup in the primary, where budget allows, secondary as well.

This is general idea only.

Cheers
 
@Bookmage

Maybe it's just me, but I'd consider Seagate last among equals (or just last :p). Their Savvio drives are excellent by all accounts, but their Barracuda and Constellation ES series aren't so good, seemingly. Personally, I've never had good experiences with Seagate; I'd go with Hitachi or Western Digital RE drives.
 
Bookmage said:
We have a similar need at work. The basic requirement is to store 40TB of contiguous data as cheaply as possible.

After comparing the time/effort I would need to build my own w/a Norco 4224 and supermicro motherboard and sas card/expander. I broke down the project into 2 parts. The storage case and the server.
For the server, I'm looking at the Dell R310 and pairing it with the Dell SAS 6 HBA or the Dell PERC H800. The difference is $600 for the raid card. The Dell R310 supports 2 x pcie 8x cards so we can use one for SAS controller and one for dual/quad GBe nic or another SAS controller.

I'm thinking of going with the NORCO DS-24E for the storage part and using the Seagate Barracuda drives for the 5 year warranty and past performance. Although, I am looking at the Hitachi drives...

Dell R310 $1200-$3000
NORCO DS-24E $1400
24 x 2TB Sge 7k drives @ $170 = $4100
or
24 x 2TB Sge 5k drives @ $100 = $2400
external SAS Cable <$100

So we can go as cheap as $1200 + $1400 + $2500 = $5000 for non raid slower drives
or $3000 + $1400 + $4200 = $8600 for raid and faster drives. The business need will determine whether we go with the better barracuda drives or just settle with the slow drives. We're also planning on keeping a couple cold spares. The real cost variable comes down to the server and with this method, we can buy cheap server now, and just swap out the server later.
We can expand this storage by adding another DS-24E + drives for $4000-$5600 or utilize 3TB drives next year.

We also looked at picking up several 2U QNAP 859+ for $2000 plus cost of drives and using ISCSI to chain them together. But that was more work and still seemed like a hack to get 40TB of contiguous space. It was $6000 to house 24 drives versus $2500-$3500 to house the same 24 drives.

I might pick up a Dell 6Gbps HBA to test with the HP SAS expander... If I really want an out of the box solution I might get the DS-24E to test with...

Let us know what you do and post a build log if you decide to build your own :-D
Click to expand...

You sound like you are in a similar situation to me however for me rack space is also at a premium so I will opt to not use a jbod expander and a separate server.

I will be going with slower drives in 2 or 3 raid6 arrays so I should be able to come close to saturating a gigabit link.
 
schnell said:
What OS are you running? Raid level? Have you had any drive failures in 12mo?
Click to expand...

OpenSolaris & OpenIndiana (latter preferred for the zpool scrub lower impact)
Two with smart errors - non fatal, so the drive was manually replaced via zfs cmd to a replacement.
7 disks per raidz stripe in one zpool for 28 disks + 2 spares
 
parityboy said:
Simply from anecdotal accounts I've read in the past - more than one person has complained about their reliability. As much as this comes from Wikipedia, it gels with what I have read in the past.
Click to expand...
Not sure I would listen to a lot of those accounts.

I have over 100 of the ST31000340NS drives all of them have been in 24/7 medium/heavy usage for several years now and only 2-3 of them have failed and I believe those 2-3 were problematic drives from the start as they had errors early on.

I will admit it was annoying having to flash all the drives with an updated firmware (I use AN05 on all of them) but after that I haven't had any issues.

I also have 48 of the ST32000444SS drives and they didn't need an updated firmware or anything special so far they have been solid going on about 4 months now under 24/7 usage.
 
You must log in or register to reply here.
Back
Top