• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Software RAID - Advice needed

O

Obiron

n00b
Joined
Dec 8, 2010
Messages
6
First post here, I just want to start by saying I really appreciate all the information available on these forums.

I recently bought components for a new WHS Vail file server (silly me, buying hardware based on a Beta OS :eek:) and with the removal of DE I've decided to go another route. At the moment, that route appears to be software RAID. I didn't buy a motherboard with RAID in mind, so it only supports 0/1/10/JBOD, and I don't want to buy a controller card at this point in time. Later down the road I intend to build a larger rack-mount server, so I'll worry about better hardware at that time. I also will probably look more into ZFS and other software at that time (unless I can be convinced that it is a better option now, I think a simple RAID 5 will be fine until my next large expansion).

The plan which currently holds the lead in my view is Linux (specifically Ubuntu Server) using mdadm. I also have Windows Server 2008 as an option, but unless anyone can show that Microsoft's software RAID solution is beneficial, I've just read too many negative comments in comparison to mdadm. So I'm open to suggestions as to an OS, but barring any I'll get on to my current setup and questions.

My server is starting with 3 2TB drives, and once it's fully running I'll transfer data off the 2 2TB drives in my main PC and grow the RAID to 5 devices. I've never done anything with software RAID, so I've formatted my server and set it up multiple times just learning and trying to decide the best design. I've read that there should be a separate swap partition, and that the OS should be on a RAID 1 because of the bootloader. I've created an 8GB partition on each drive for swap, 100GB for OS, and the remainder for storage. I've set up a mirror array on the first two drives for the swap, a second mirror for OS, and a RAID 5 across all drives for storage. It looks like this:

Code: 
   sda           sdb          sdc
----------   ----------   ----------
|    1    |  |    1    |  |    x    |
----------   ----------   ----------
|         |  |         |  |         |
|    1    |  |    1    |  |    x    |
|         |  |         |  |         |
----------   ----------   ----------
|         |  |         |  |         |
|         |  |         |  |         |
|         |  |         |  |         |
|         |  |         |  |         |
|    5    |  |    5    |  |    5    |
|         |  |         |  |         |
|         |  |         |  |         |
|         |  |         |  |         |
|         |  |         |  |         |
|         |  |         |  |         |
----------   ----------   ----------

This leaves unused space on the third drive, which will also carry over to the other two drives that I will add in, looking like this:

Code: 
   sda           sdb          sdc          sdd          sde
----------   ----------   ----------   ----------   ----------
|    1    |  |    1    |  |    x    |  |    x    |  |    x    |
----------   ----------   ----------   ----------   ----------
|         |  |         |  |         |  |         |  |         |
|    1    |  |    1    |  |    x    |  |    x    |  |    x    |
|         |  |         |  |         |  |         |  |         |
----------   ----------   ----------   ----------   ----------
|         |  |         |  |         |  |         |  |         |
|         |  |         |  |         |  |         |  |         |
|         |  |         |  |         |  |         |  |         |
|         |  |         |  |         |  |         |  |         |
|    5    |  |    5    |  |    5    |  |    5    |  |    5    |
|         |  |         |  |         |  |         |  |         |
|         |  |         |  |         |  |         |  |         |
|         |  |         |  |         |  |         |  |         |
|         |  |         |  |         |  |         |  |         |
|         |  |         |  |         |  |         |  |         |
----------   ----------   ----------   ----------   ----------

Would it be advisable to mirror the swap and OS across all 5 drives? Or maybe set up another RAID 5 across the last three 100GB partitions? Or just leave those unused? Or am I just way off the mark? Normally I would put the OS on a small drive separate from the storage, but my motherboard only has 5 SATA slots and I have 5 large drives to use. I want to use my space efficiently, but if my current thought process is a bad idea, then I'm open to better suggestions of how to make the best use of my 5 drives.

If I need to provide any more info about my hardware, or anything concerning my setup, please let me know. I appreciate any help I can get with this.
 
Do you really need 100G for OS? On my servers I rarely use more than 3G. I'd make a 10G partition were I you.

Why not put swap in the spare 100G partition on the third drive? I think this setup should be fine, although if I were in your position I'd just buy a pci-e sata card and use two separate RAID1 drives for OS. Make sure you play with chunksize performance on your platform before you move files over because it takes ages to reshape chunksize.

Otherwise seems fine. Running 10.04.1?
 
Have you looked at flexraid? Its a good option if you go with windows server 2008
 
Would it be advisable to mirror the swap and OS across all 5 drives?
Click to expand...

At work I usually create a separate swap on each drive of a raid array. And since there are many of them 512MB to 1G would be fine. I do however create a separate boot from / and raid 1 that accross all drives in the array. As for / 100G is huge. On a binary distribution like ubuntu I would cut that to 10G. On a source based distro like gentoo or Arch I would use 40 to 50G since they tend to need more space. Also I usually make / raid 5 or 6 and use the 3rd partition for that. With the first being /boot and the second being swap.

Then in the 4th partition (on all drives) I create a raid 5 or 6 array. Inside this raid 5/6 array for data I use lvm so that I can subdivide my data into each project and grow and shrink at will (filesystem like ext4 needed for that instead of xfs).

Also if I was not clear. In my partition scheme I make each disk have the exact same layout. I only partition 1 drive and use sfdisk to copy the partion table to the others

sfdisk -d /dev/sda | sfdisk /dev/sdb

I also take the time to install grub on every single disk. Since my boot is a raid 1 with all disks this will allow any disk at all to be the boot disk.


Here is one older example using 6 320 GB drives in raid
Code: 
datastore0 ~ # sfdisk -d /dev/sda
# partition table of /dev/sda
unit: sectors

/dev/sda1 : start=       63, size=   514017, Id=fd, bootable
/dev/sda2 : start=   514080, size=  1510110, Id=82
/dev/sda3 : start=  2024190, size= 23454900, Id=fd
/dev/sda4 : start= 25479090, size=599642190, Id=fd

datastore0 ~ # fdisk -l /dev/sda

Disk /dev/sda: 320.1 GB, 320069031424 bytes
255 heads, 63 sectors/track, 38912 cylinders, total 625134827 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *          63      514079      257008+  fd  Linux raid autodetect
/dev/sda2          514080     2024189      755055   82  Linux swap / Solaris
/dev/sda3         2024190    25479089    11727450   fd  Linux raid autodetect
/dev/sda4        25479090   625121279   299821095   fd  Linux raid autodetect


datastore0 ~ # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] [multipath]
md0 : active raid1 sdf1[5] sdd1[3] sde1[4] sdb1[1] sdc1[2] sda1[0]
      256896 blocks [6/6] [UUUUUU]

md2 : active raid6 sdf4[5] sdd4[3] sde4[4] sdb4[1] sdc4[2] sda4[0]
      1199283200 blocks level 6, 256k chunk, algorithm 2 [6/6] [UUUUUU]

md1 : active raid6 sdf3[5] sdd3[3] sde3[4] sdb3[1] sdc3[2] sda3[0]
      46909440 blocks level 6, 64k chunk, algorithm 2 [6/6] [UUUUUU]

unused devices: <none>

datastore0 ~ # lvs
  LV              VG   Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  distfiles       vg0  -wi-ao  70.00g
  ds0_image_space vg0  -wi-ao 600.00g
  user_private    vg0  -wi-ao 200.00g
  user_public     vg0  -wi-ao 200.00g
  vservers        vg0  -wi-ao  68.72g
datastore0 ~ # pvs
  PV         VG   Fmt  Attr PSize PFree
  /dev/md2   vg0  lvm2 a-   1.12t 5.00g

datastore0 ~ # swapon -s
Filename                                Type            Size    Used    Priority
/dev/sda2                               partition       755048  168     -1
/dev/sdb2                               partition       755048  0       -2
/dev/sdc2                               partition       755048  0       -3
/dev/sdd2                               partition       755048  0       -4
/dev/sde2                               partition       755048  0       -5
/dev/sdf2                               partition       755048  0       -6

datastore0 ~ # mount | grep md
/dev/md1 on / type reiserfs (rw,noatime)
/dev/md0 on /boot type ext2 (rw,noatime)

datastore0 ~ # mount | grep mapper
/dev/mapper/vg0-distfiles on /mnt/vg0/distfiles type reiserfs (rw,noatime)
/dev/mapper/vg0-user_private on /mnt/vg0/user_private type reiserfs (rw,noatime,acl,user_xattr)
/dev/mapper/vg0-user_public on /mnt/vg0/user_public type reiserfs (rw,noatime,acl,user_xattr)
/dev/mapper/vg0-ds0_image_space on /mnt/vg0/ds0_image_space type reiserfs (rw,noatime)
/dev/mapper/vg0-vservers on /vservers type reiserfs (rw,noatime)

This is a 4 year old system so I am still using reiserfs there. I do not do that on new systems. I generally use ext4 for everything except /boot now.
 
Last edited:
@tgrimley: No, 100G isn't really necessary. I started with that just for testing and haven't changed it yet. You're probably right, and in the end I'll make that partition smaller (I'm sure I'll format a couple more times before I actually transfer data over). I hadn't thought of putting swap on the third drive, but that's a good idea. I guess that doesn't really need to be mirrored, does it? I'll look at pci-e cards, that hadn't crossed my mind since I was thinking of full blown RAID cards, but that should be doable cost-wise. And I wasn't aware of chunk-size issues, so I'll look into that.
And yes, 10.04.1. I figure LTS is a good idea since I don't know when my next upgrade will be.

@Ncage: I haven't looked at flexraid. I installed Server 2008 once, and was impressed by how easy it is to set up RAID in disk management, but I was turned off by the negative reviews that I had been reading. I'll look into flexraid and keep that in my list of options.

@drescherjm: That's interesting regarding GRUB. This is actually where I've had some issues so far. I set up my RAID arrays during the install (where you're copying the partition table, I assume you're doing this prior to install? Or after?) and my biggest issue has been that GRUB won't install at the end. I can get LILO to install though so I can boot into the OS. From there, I've been working on changing to GRUB. I think I've got it working, but I won't know until I get home because I can't VNC after my latest reboot (I have Ubuntu desktop installed). I can SSH though, so it seems that everything is up and running.
 
Last edited:
You must log in or register to reply here.
Back
Top