mdadm RAID5 ate my drives!

O

outspoken

Limp Gawd
Joined
Feb 15, 2007
Messages
465
Does this seem right to anyone? I just put in 2 3TB drives into my RAID5 and was only given back 1.8TB of usable space.
Code: 
/dev/md0                      6.3T  4.5T  1.8T  72% /data

Code: 
/dev/sdh1 - 1TB (capacity: 931GB)
/dev/sdi1 - 2TB (capacity: 1863GB)
/dev/sdg1 - 1TB (capacity: 931GB)
/dev/sdf1 - 1TB (capacity: 931GB)
/dev/sde1 - 1TB (capacity: 931GB)
/dev/sdd1 - 2TB (capacity: 1863GB)
/dev/sdb1 - 3TB (capacity: 2047GB)
/dev/sda1 - 3TB (capacity: 2047GB)

Code: 
mdadm -D /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Mon Feb 13 23:06:17 2012
     Raid Level : raid5
     Array Size : 6837315072 (6520.57 GiB 7001.41 GB)
  Used Dev Size : 976759296 (931.51 GiB 1000.20 GB)
   Raid Devices : 8
  Total Devices : 8
    Persistence : Superblock is persistent

    Update Time : Mon Mar 10 08:00:56 2014
          State : clean 
 Active Devices : 8
Working Devices : 8
 Failed Devices : 0
  Spare Devices : 0
         Layout : left-symmetric
     Chunk Size : 512K

           Name : silvermedia:0
           UUID : 729ba0a4:d949b840:b9a96d7f:f9e9447f
         Events : 40891

    Number   Major   Minor   RaidDevice State
       0       8       97        0      active sync   /dev/sdg1
       6       8       81        1      active sync   /dev/sdf1
       2       8       65        2      active sync   /dev/sde1
       4       8      129        3      active sync   /dev/sdi1
       5       8      113        4      active sync   /dev/sdh1
       7       8       49        5      active sync   /dev/sdd1
       9       8       16        6      active sync   /dev/sdb
       8       8        0        7      active sync   /dev/sda
 
RAID5 in mdadm using raw disks uses the lowest common denominator of disk size (eg the smallest disk size is how large all the per disk usage will be.) Even though you added 2 3TB drives, for all intents and purposes you added just 1TB drives as far as mdadm is concerned in RAID5 using raw disk mount points.
 
If you mix drive sizes like that, you're only going to be using the capacity of the smallest drive for each drive. In this case, you're only getting about 1TB per drive, minus a drive for parity (hence the 7TB array size). Even the 2TB drives were being under-utilized prior to adding the 3TB ones.
 
Let alone that you seem to be unable to even fully use 3TB drives, judging from

/dev/sdb1 - 3TB (capacity: 2047GB)
/dev/sda1 - 3TB (capacity: 2047GB)
Click to expand...
 
Correction, RAID5 ALWAYS uses the lowest common denominator, not just the mdadm implementation. That's the defintion of RAID5. Maybe you should've started a new RAID5 and then created a new volume across those devices.. :)
 
FrankD400 said:
Correction, RAID5 ALWAYS uses the lowest common denominator, not just the mdadm implementation. That's the defintion of RAID5.
Click to expand...
What? This is news to me. Do you have any links or more information on this? This sounds too weird to be true?
 
What is weird about it? If data is going to be striped over the N drives, how else would you handle different size drives?
 
brutalizer, not much I can say other than look up RAID5 and how it works :) You could not make use of extra space and have the parity information with the way that RAID5 stripes data/parity across the drives. Any implementation with N+1(parity) that allowed this would not be remotely close to being RAID5.

How does is sound too weird to be true? Any other answer is too weird to be true. :)
 
I believe you are thinking about some of the non-standard raids like "unraid" which allow things like oddly sized disks to be "raided" together.

As others have stated, Standard raid-5 is a strip of fixed size across N drives. if you take three 1 TB drives and 1 2-TB drive and raid-5 them the 2-tb drive is considered a 1TB drive for the raid. I would guess the rest of the space would be available as a non-raid partition on the disk.


brutalizer said:
What? This is news to me. Do you have any links or more information on this? This sounds too weird to be true?
Click to expand...
 
TCM2 said:
Let alone that you seem to be unable to even fully use 3TB drives, judging from
Click to expand...

oh no.... so what's the problem here then?

Code: 
     *-scsi:0
          physical id: 1
          logical name: scsi2
          capabilities: emulated
        *-disk
             description: ATA Disk
             product: TOSHIBA DT01ACA3
             vendor: Toshiba
             physical id: 0.0.0
             bus info: scsi@2:0.0.0
             logical name: /dev/sda
             version: MX6O
             serial: 13T9M9MAS
             size: 2794GiB (3TB)
             capabilities: partitioned partitioned:dos
             configuration: ansiversion=5 sectorsize=4096 signature=730838e4
           *-volume UNCLAIMED
                description: EFI GPT partition
                physical id: 1
                bus info: scsi@2:0.0.0,1
                capacity: 2047GiB
                capabilities: primary nofs
     *-scsi:1
          physical id: 2
          logical name: scsi3
          capabilities: emulated
        *-disk
             description: ATA Disk
             product: ST3000DM001-1CH1
             vendor: Seagate
             physical id: 0.0.0
             bus info: scsi@3:0.0.0
             logical name: /dev/sdb
             version: CC49
             serial: Z1F45SLS
             size: 2794GiB (3TB)
             capabilities: partitioned partitioned:dos
             configuration: ansiversion=5 sectorsize=4096 signature=b55d9085
           *-volume UNCLAIMED
                description: EFI GPT partition
                physical id: 1
                bus info: scsi@3:0.0.0,1
                capacity: 2047GiB
                capabilities: primary nofs

This is the device they are connected to:

Code: 
05:04.0 RAID bus controller: Silicon Image, Inc. SiI 3124 PCI-X Serial ATA Controller (rev 02)

I'm guessing the best course of action is to remove the 1TB drives completely.
 
You may just want to move to snapraid for this assortment of drives. Also I would switch to double parity. raid5 is not good for that many drives.

As for the SIL PCI-X raid chip. I doubt that has support for >2TB drives.
 
drescherjm said:
You may just want to move to snapraid for this assortment of drives. Also I would switch to double parity. raid5 is not good for that many drives.

As for the SIL PCI-X raid chip. I doubt that has support for >2TB drives.
Click to expand...

crap. it seemed to be the best for the price at MicroCenter. not many stores stock sata/raid controllers on shelves =)
 
outspoken said:
crap. it seemed to be the best for the price at MicroCenter. not many stores stock sata/raid controllers on shelves =)
Click to expand...

You can buy Dell H310 controllers for like $60 on eBay. Just flash to IT firmware and it makes a fine HBA.
 
dexvx said:
You can buy Dell H310 controllers for like $60 on eBay. Just flash to IT firmware and it makes a fine HBA.
Click to expand...

I'll just switch out the two 3TB and put them on the mobo sata, that shouldn't cause any issues right? putting the 1TB on the other controller.
 
Or, do like synology does with its SHR

Use madam on top of lvm

So 1tb drives have one logical volume of 931gb
2tb drives have 2 logical volumes of 931gb
3tb drives naturally have 3 logical volumes of 931gb

So then you raid5 the 1st 931gb volume on each of the 8 drives (1tb + 2tb + 3tb) as another volume
Then RAID 5 the 2nd volume of 931gb on the 2tb and 3tb drives into a volume
And again for the remaining 3tb drives.... Either RAID 5 in a degraded state... Or as a mirror..

Then bunch the raided volumes together.
Eg, have a look see here

https://www.synology.com/en-us/support/RAID_calculator

Although synology does t all behind the scenes... And you doesn't have to work it out manually.;)

.
 
Stanza33 said:
Or, do like synology does with its SHR

Use madam on top of lvm

So 1tb drives have one logical volume of 931gb
2tb drives have 2 logical volumes of 931gb
3tb drives naturally have 3 logical volumes of 931gb

So then you raid5 the 1st 931gb volume on each of the 8 drives (1tb + 2tb + 3tb) as another volume
Then RAID 5 the 2nd volume of 931gb on the 2tb and 3tb drives into a volume
And again for the remaining 3tb drives.... Either RAID 5 in a degraded state... Or as a mirror..

Then bunch the raided volumes together.
Eg, have a look see here

https://www.synology.com/en-us/support/RAID_calculator

Although synology does t all behind the scenes... And you doesn't have to work it out manually.;)

.
Click to expand...

Came here to say this too. MDADM can do RAID5 or RAID6 with different sized disks and utilize all or nearly all the space in most setups using LVM to help.

That Synology is sure some smart software, but you can set up what it does yourself too.
 
At work I used to use lvm on top of mdadm (sometimes spanning more than array) and ext4 or btrfs on top of that but these days I am migrating all of this to zfs on linux. In that case I would create a pool with more than 1 vdev each vdev containing the same sized disks arranged as mirrors, raidz1 or raidz2.
 
Stanza33 said:
Or, do like synology does with its SHR

Use madam on top of lvm

So 1tb drives have one logical volume of 931gb
2tb drives have 2 logical volumes of 931gb
3tb drives naturally have 3 logical volumes of 931gb

So then you raid5 the 1st 931gb volume on each of the 8 drives (1tb + 2tb + 3tb) as another volume
Then RAID 5 the 2nd volume of 931gb on the 2tb and 3tb drives into a volume
And again for the remaining 3tb drives.... Either RAID 5 in a degraded state... Or as a mirror..

Then bunch the raided volumes together.
Eg, have a look see here

https://www.synology.com/en-us/support/RAID_calculator

Although synology does t all behind the scenes... And you doesn't have to work it out manually.;)

.
Click to expand...

Using mdadm on top of LVM is just asking for trouble. And these multilayered complex setups in general tend to be fragile if only a single component has a small hiccup.
I started with XFS on LVM on dmcrypt on mdadm on GPT, transitioned to XFS on dmcrypt on mdadm and now finally use ZFS on dmcrypt.
 
Last edited:
Doing multi layered is ok. Just make sure everything aligns:

physical disk (512b ? 4k ? 128k or 1M for ssds?)
mdadm (# of usable disks should be 2^y = usable disks.....so 2/4/8/16....)
lvm2 (proper pe size & pv alignment)
filesystem formatting parameters...
If you are using ssds, make sure trim (can) pass through all layers.

With that being said, I use ZFS on FreeBSD as it's kernel level (not sure if linux is still fuse or not?). it's also been in FreeBSD for...7'ish years now? Works great on an athlon ii 265 (IIRC) with 8G ram ecc on m5A99x pro 2 r2
 
drescherjm said:
zfsonlinux is a native kernel level module. Performance is much better than the fuse module. Also with the Lawrence Livermore National Laboratory behind the port development progresses faster.

http://zfsonlinux.org/
Click to expand...

I'll take a look, thanks for the info.

Any comparisons (esp on featureset/stability) to the FreeBSD version?
 
Has most of the most recent code changes. If you go with that, use trueos (freebsd server distro), since it gives you be (boot environments).
 
ZFS on Linux is very up to date with Illumos and FreeBSD as far as features go (v5000 / v5 and v5000 feature flags.)

Where ZoL is behind is in performance and stability under certain conditions (using more advanced features (zvols and heavier loads)). For a home media server for though example ZoL is great. I have been using it 6 months now and not a single issue whatsoever.
 
If you need a PCI-X card I can recommend this:

Supermicro AOC-SAT2-MV8
http://www.newegg.com/Product/Product.aspx?Item=N82E16815121009

I have actually used them with 3 TB drives on linux with no issues. I hear that the supermicro drivers on windows have issues but there are some alternate drivers you can use. Here is some performance testing I did back in the day on a machine with a PCI-X slot:

http://hardforum.com/showthread.php?t=1388900

I actually have a few laying around if you would like I can send you one if you pay for shipping and PM me your address. It might take a few days for me to ship one though.

You can get them pretty cheap on ebay as well $30 and under.
 
I just bought 2 more 3TB drives and I'm looking to remove all of the 1TB drives from the setup. My plan:

  • Use FlexRAID to create a pool with the 2 new 3TB drives
  • Copy over all of the data from /dev/md0 to the new pool.
  • Delete the mdadm RAID5 array.
  • Add remaining 2TB and 3TB to the pool.
 
DO NOT use raid 5, of all the raids to use... Raid 5 is the worst this day in age!!!

Raid 6 for storage raid 10 for performance.
 
1TB drives are actually about 900GB.

When they say TB they are calculating it as 1,000,000,000,000 bytes when in reality a TB is 1,099,511,627,776 bytes. So you "lose" 99,511,627,776 bytes which is about 99GB. Adds up quick.

My math might be off, it's still morning. :p
 
Ok, but I think I'm past that point. I understand 1TB=931GB, it's shown in the original post. I didn't know about RAID reducing to the smallest amount available, so that's why I'm now going to move to to a drive pool solution like FlexRAID.

If anyone has suggestions on the implementation or can verify what I've outlined above will work I would appreciate it.

Thanks to all of you for the great info.
 
Last edited:
Oh yeah that's completely normal since raid needs disks the same size, so if one is smaller it has to treat the others as that size. Best bet is make separate arrays using same size disks.

With mdadm raid you can make partitions and raid the partitions but don't know what the performance would be like. Ex:

Drive A: 2TB
drive B: 1TB
drive C: 1TB

You could make two 1TB partitions on drive A, leaving you with a total of 4 1TB partitions then raid those.

I THINK it can be done anyway, never done it myself, or recommend it.
 
No, 1 TB is 1,000,000,000,000 bytes. You are thinking about 1 TiB which is 1,099,511,627,776 bytes.

1 TB = 1000 GB = 931.3 GiB.
 
You must log in or register to reply here.
Back
Top