Seeking advice on RAID setup for my shiny new Linux server

MountainX

Limp Gawd
Joined
Mar 2, 2008
Messages
254
I just burned a CD for Ubuntu Hardy (8.4.1 AMD 64 server) and I plan to install it on my home file server. I'm ready to get started now, but I realized I need advice on how to set the RAID controllers (cache IO or not, writeback or writethru, read ahead or not, stripe size, disk spin down, etc.).

My hardware is:
Dual Xeon with 6 GB RAM

LSI MegaRAID 320-2E SCSI controller with:
4x Seagate ST373454LC 15k rpm drives.

logical drives:
1. two physical disks as RAID 1 for Ubuntu (boot, root)
2. one physical drive for swap, application scratch, logs, etc.
3. one physical drive for virtual machine (dedicated) or other uses

Areca ARC-1220 SATA PCI-express 8x controller with:
4x Samsung HD103UJ
2x WD740GD Raptors

The raptors are set up as passthrough only. They will be dedicated to virtual machines or other things.

The 4x Samsungs are the main storage. I plan to use RAID 5 with one 2 TB volume and a 1 TB volume.

I would appreciate any advice on how to set up the RAID for this home file server. This is awesome hardware for my situation and I'd like to get the most out of it.

All computers are connected via GigE. All our family data will reside on this file server. This includes music, photos, documents, everything. Most folders in my home partition are mounted folders that reside on this file server. (At the moment, we have a Windows-based file server that will be replaced by this one.) So I need really snappy performance from the network and file server in order not to notice that all my documents are not on my local computer.

BTW, I have an older computer that is used to back up the data from the file server. (Not doing offsite backup.)

I also think it might be good to spin down the drives after inactivity because the file server runs 24 x 7, but when the family is asleep why not save some wear and tear on the disks? Advice on this?

And I'm not an admin - maybe you could call me a power user. I'm will be happy if replies are detailed/elaborate. Otherwise I might not understand.
 
Obviously I don't know what your use cases are going to be like but that is pretty ludicrous hardware for a home file server.

For me personally I like to have a minimal number of hard drives and controller cards since the more hardware in play the more failures you will have to deal with. But you may have specific goals I don't know about.


Some miscellaneous notes:

With that much memory you are probably never going to touch the swap file so it may not be worth having an extra drive in there just for that. With most servers that are just running linux/samba/nfs/cups etc you would be lucky to even use half a gig of memory, and that's assuming you install a gui like kde or gnome. You mention virtual machines so I guess that could use any amount of memory but you would still have to have some strange use cases for a home server to pass 6gigs.

In my opinion (some people will disagree with this) if the machine is going to be accessed on a regular basis it is not worth spinning down the drives, and could easily do more harm that good.

In the vast majority of cases I am imagining your network is going to be the bottleneck, not the drives.


What I would probably do (again not knowing the specifics of your situation):

if you are going to be doing something live having people mount their home directories in linux via nfs, or their documents and settings directories via samba, and latency is important you could do one of the following:
put the 4 scsi drives and the scsi card in a raid10
or
put the two sata raptors in a linux software raid1

and for mass storage I would put the 4 samsung 1TB drives into a linux software raid5 (leaving out the sata controller entirely if you have enough sata ports) and have another one of those drives onhand or as a hot spare if one fails, or just use 3 of the drives in the active raid if you don't want to get an additional one.


I personally choose linux software raid over hardware raid almost any chance I get, and I have used a variety of hardware raid cards in the past. This is something that a lot of people don't like to hear, especially those who built servers through the 90s when software raid was a lot worse and cpus were a lot slower, but in my opinion software raid has a lot of real advantages, such as:
You have almost unlimited flexibility with respect to things like block sizes, raid type, multiple raid configurations on the same disk, etc.

It is well tested and widely used and will continue to be updated automatically, whereas bugs in hardware controllers will only be fixed as long as the manufacturer feels like putting out firmware updates.

The on-disk format is non-proprietary and widely used and understood, if the motherboard fails you can pull the drives out and throw them in another linux machine and get at your files. If a hardware controller fails you sometimes have to find a similar controller, preferably running the same firmware, or use special software which doesn't always work.
 
Obviously I don't know what your use cases are going to be like but that is pretty ludicrous hardware for a home file server.

:D yes, it is.

For me personally I like to have a minimal number of hard drives and controller cards since the more hardware in play the more failures you will have to deal with. But you may have specific goals I don't know about.

Not really. This is more of a situation where I have more hardware than I need but I'd rather make use of it than leave it sitting in the corner. And I will use the heck out of it. The Windows file server I set up several years ago was also overkill (2 TB RAID, 4 GB RAM, dual Xeon, etc.) and that thing has been great. It has been wonderful to have all our data in one place on reliable hardware.

Some miscellaneous notes:

With that much memory you are probably never going to touch the swap file so it may not be worth having an extra drive in there just for that.

OK. I have noticed on my desktop machine with 4 GB of RAM running Ubuntu and Compiz that the swap file is almost never touched.

With most servers that are just running linux/samba/nfs/cups etc you would be lucky to even use half a gig of memory, and that's assuming you install a gui like kde or gnome.

I am going to install gnome.

You mention virtual machines so I guess that could use any amount of memory but you would still have to have some strange use cases for a home server to pass 6gigs.

I'm also going to install vmWare or VirtualBox and run a couple virtual machines -- but they won't be running all the time.

In my opinion (some people will disagree with this) if the machine is going to be accessed on a regular basis it is not worth spinning down the drives, and could easily do more harm that good.

By regular basis do you mean daily? So you would not spin down the hard drives while the computer isn't being used overnight?

Actually, the max time I can set for spin down is 60 minutes of inactivity.

In the vast majority of cases I am imagining your network is going to be the bottleneck, not the drives.

My network is GigE with a Netgear GS716T switch. On our network I have seen times when the drives were the limiting factor in copying video files -- but that should not be the case with this hardware.

What I would probably do (again not knowing the specifics of your situation):

if you are going to be doing something live having people mount their home directories in linux via nfs, or their documents and settings directories via samba, and latency is important you could do one of the following:
put the 4 scsi drives and the scsi card in a raid10
or
put the two sata raptors in a linux software raid1

and for mass storage I would put the 4 samsung 1TB drives into a linux software raid5 (leaving out the sata controller entirely if you have enough sata ports) and have another one of those drives onhand or as a hot spare if one fails, or just use 3 of the drives in the active raid if you don't want to get an additional one.

I personally choose linux software raid over hardware raid almost any chance I get, and I have used a variety of hardware raid cards in the past. This is something that a lot of people don't like to hear, especially those who built servers through the 90s when software raid was a lot worse and cpus were a lot slower, but in my opinion software raid has a lot of real advantages, such as:
You have almost unlimited flexibility with respect to things like block sizes, raid type, multiple raid configurations on the same disk, etc.

It is well tested and widely used and will continue to be updated automatically, whereas bugs in hardware controllers will only be fixed as long as the manufacturer feels like putting out firmware updates.

The on-disk format is non-proprietary and widely used and understood, if the motherboard fails you can pull the drives out and throw them in another linux machine and get at your files. If a hardware controller fails you sometimes have to find a similar controller, preferably running the same firmware, or use special software which doesn't always work.

If I didn't already have this hardware, I would consider your approach. But I don't think I could get all these drives hooked up directly to the motherboard and I like the hotswap + autorebuild capabilities I get by using these controllers. And again, what would I do - let the controllers sit there and gather dust? I have them, so I would like to put them to use.

I am going to mount the home directories in linux via nfs, so latency is important. The SCSI drives are 15k rpm, so they should be a lot faster than the raptors.

I take it from your advice that using those fast SCSI drives for mounted home directories would be better than using them for swap, tmp and var. I could use all 4 SCSI drives for users home directories and set it up as RAID 1.

I could put the root partition (OS, etc.) on the 2 raptors set it up as RAID 1.

I would probably use write through, adaptive read, and IO caching settings.

How does that sound?
 
With regard to what to do with the unused hardware in my previous suggestion, I was thinking to sell it. But if you are determined to use all that crazy stuff one thing you could do:

the two raptors in a raid1 containing the swap partition and everything for the os except /home

the four seagate 15k-rpm drives in a raid10, with one partition just holding /home

the four samsung 1TB drives in a raid5 with one giant partition that you could mount at /media/share or somesuch place, for all the big files you want everyone to access.


For file system stuff I would probably use ext3 and set noatime (a checkbox when doing partitioning) for all the partitions. Ext3 performance isn't as great as some other filesystems such as xfs but ext3 is super safe, will be supported forever. The only times you will really notice waiting for it is when you are deleting tens or hundreds of gigs at once.

If you have never set up nfs4 before it can be a bit of a pain but very high performance. Assuming this is only for internal use by trusted users you can probably disable most of the security stuff and make the setup a lot easier. There is a howto on the ubuntu wiki somewhere iirc.

Also with that many drives and using lots of caching I hope you have that thing on a ups.
 
Thanks again for your reply. Your comments are helpful. As to selling the hardware, I would only get pennies on the dollar and I actually prefer hardware raid anyway, so I definitely want to use what I've got.

I like your suggestion to set up the 2 raptors as raid1 for os, swap and everything except /home.

I think I'll set up the four seagate 15k rpm drives in a raid5 (instead of raid10) and put /home on there as you suggested. (I wouldn't use raid5 if I were using software raid.)

I'll do exactly as you suggested with the four samsung 1TB drives.

Thanks for the tips on noatime and on ext3. I'll use both those suggestions. I'll also use write through, adaptive read, and cache I/O settings on the hardware raid controllers.

Not only do I have everything on multiple UPS's, I also have a generator. But why do you feel I'm using lots of caching? I'm not using write back caching and I am not aware of what other settings could create problems. Should I use direct I/O? What affect will that have on latency?

The only time I have set up nfs on Ubuntu I believe I used nfs3. I will use nfs4 this time -- but I'll probably take this opportunity to learn about the security settings. I'm probably going to set up a personal WebDAV server (using only davs://) so I can get to my files while I'm away from home, so I don't want to take chances on security given that the file server will be attached to the Internet.

Thanks again.
 
Alright good luck with the nfs setup. Nfs3 was pretty weak from a security standpoint so they went kind of overboard with nfs4. It's used widely in corporate installations where they have everyone with their home directory mounted via nfs on a server, but setting it up for home use there is a bit of overhead. So if you want to enable all the security stuff be prepared to do a bit of reading.

On accessing files remotely, a super easy way to do that is using ssh (sftp). If you are accessing it from Linux or a mac there is sshfs so you can view your remote machine just as a normal mounted filesystem. If you are using a windows machine there are some programs to map a drive to an sftp share but in my experience they are pretty buggy leaving webdav as a possible better option.
 
Thanks again. If NFS4 is that much more complex, I may have to simplify my settings...

The neat thing about WebDAV is that all the clients (Win, Linux, Mac) can access it. I am usually forced to run Windows on my laptop (although I wish that wasn't the case).
 
Do I need to have any reserved blocks for the super-user on the volumes that are used for mass storage? I left 5% reserved blocks on the volume for root (also has /tmp, /var, /usr, etc.) and on the boot partition.

The home partition (3TB storage space) has 0% reserved blocks. Is this OK?
 
I'm not 100% sure on this but i think the only point of reserve space is to allow the filesystem to minimize fragmentation when the drive gets close to full, and to ensure there is space avaiable in /tmp and /var for temp files, log files, and the like.

"
man tune2fs
-m reserved-blocks-percentage
Set the percentage of the filesystem which may only be allocated by privileged processes. Reserving some number of filesystem blocks for use by privileged processes is done to avoid filesystem fragmentation, and to allow system daemons, such as syslogd(8), to continue to function correctly after non-privileged pro‐cesses are prevented from writing to the filesystem. Normally, the default percentage of reserved blocks is 5%.
"

So I think if that giant raid5 partition has no os directories on it you are ok without reserved blocks, you will have to remember to watch free space remaining to avoid fragmentation rather than have the filesystem fence off that space in advance. But fragmentation with an almost full disk is an issue with any filesystem so that's not a big deal.
 
I'm not 100% sure on this but i think the only point of reserve space is to allow the filesystem to minimize fragmentation when the drive gets close to full, and to ensure there is space avaiable in /tmp and /var for temp files, log files, and the like.

Reserve space is there so you can still log in if you fill your drives. If you /completely/ fill a drive, I don't think you can log in as anything other than (maybe) superuser, and you have to truncate files (can't just delete them) to free some space up. This does depend somewhat on the filesystem you are using, as some are smarter about it than others, but it's a good idea nonetheless.

You may want to reserve 1-2% on the home partition, because when the disk starts to fill up your performance will suffer a lot if you don't.
 
All computers are connected via GigE. All our family data will reside on this file server. This includes music, photos, documents, everything. Most folders in my home partition are mounted folders that reside on this file server. (At the moment, we have a Windows-based file server that will be replaced by this one.) So I need really snappy performance from the network and file server in order not to notice that all my documents are not on my local computer.

You don't need all of your hardware or linux to accomplish this. That's serious overkill.

I do this with my WHS box and hardware that is 5 years old. An old 2.4ghz Pentium 4, 2gb of ram and an old Asus board with some Promise TX4 sata add in cards in PCI slots.

I mount my photos, music and documents folders in Vista and Leopard and access that data as if it were local with no problems or performance issues. And I stream HD video across the network to my HTPC. You would never know all my data is accessed over a network unless I told you.

For your situation maybe look into unraid.
 
Actually, depending what you are using, that hardware is not excessive at all. Look at ZFS requirements for example. I have seen on mailing lists that 32-bit may possibly not be supported. 1-2G ram MINIMUM is required. etc etc.

Also, if you ran something like lustre/samba combo (samba as of 3.2 I believe officially supports this) or opensolaris/freebsd/(eventually linux)zfs, thoat hardware is not highend.
 
rogue_jedi: yeah you're right, I forgot about that one, if /home has no space available you can't log in. Or I should say you can log in right before your display manager chokes halfway and hangs.

Archer75: he said he has all the hardware already, so might as well take a ride on the crazy train.
 
As another Linux user, I would recommend looking at something besides ext3. I have had numerous clients have the jornal blow out (recoverable, but a pain as you have to overwrite the journal, fsck it as ext2, then rebuild the journal and re-mount as ext3) and I have had a few that have completely lost filesystems. I have seen more downtime from ext3 than ext2. There is also an issue with sparse file creation with samba when writing large files from windows hosts in ext3. It has no impact on xfs, but it hurts ext3 performance really bad...

Also, if you want to use the online expansion capabilities of your RAID controller, consider not creating a partition for your /home and just creating the filesystem on the raw block device (I.E. /dev/sdb). That way if you do an online expansion, you don't have to worry about fooling around with a partition, you can just grow your filesystem.
 
xfs does have better performance in a lot of cases, my biggest worry with it is recovery when something goes wrong since it is not going to be supported indefinitely (may not be an issue here). And it has issues with data loss during power failure, Andrew Morton talks about it a little in a Google Video presentation (25min in and 59min in). But it may be suitable here, I know sony uses xfs on at least some of their dvr units, probably to avoid ext3 delete lag, so they have some confidence in it, I'm always paranoid when it comes to data loss though.

There is no perfect solution at the moment, we have to limp along with ext3 on most systems for a while, ext4 will be a stopgap. Hopefully btrfs can get to a mature state over the next couple years cause we need it, it probably will since oracle is funding it but when it gets into mainline is anyone's guess.
 
You may want to reserve 1-2% on the home partition, because when the disk starts to fill up your performance will suffer a lot if you don't.

Is it possible to change that after the fact?
(btw, /home is on an encrypted drive.)
 
Back
Top