Resizing ext4 filesystem > 16TB

I think the best option would be to look at other filesystems such as xfs or zfs... Unfortunately that means backing up everything and reformatting.

e2fsprogs still does not support >16TB officially and any of the beta releases are buggy as hell. Not something I would trust with my data. It's been a known issue for years so I wouldn't wait for them to release a stable version.

 

Might want to take a look at zfs. Supposedly the zfs on linux project is stable now. If not, you could maybe move to OpenIndiana or SolarisExpress11 with zfs.

 

Anyway you look at it, at present you're going to need to look at option 4 at the moment unless you have the ability to wait until they have the issues with EXT4 sorted out and proven.

If you can't I'd recommend JFS as a suitable filesystem as its often overlooked down to it not being actively developed by IBM. The reason however for that is different from the norm however, as from what I understand, it is that they don't need to add any further features but are still very much in the development game with it.

Houkouonchi uses it on this very large arrays with no issues and I also do on my (rather small) 18tb RAID-6 array and I've never had any problems with it so far. A few others also do, the most read about example being Backblaze on their Petabyte storage platform.

 

Another alternative to reformatting would be to convert to Btrfs.

Although with that option you will still need to wait for the fsck tool, which should be available very soon, possibly sometime in September.

 

Another option would to create another LVM and then pool the two volumes using GreyHole. This will present a single share over the network, but obviously they will still be two separate volumes on the server itself.

Since (I assume) your PVs are in fact RAID devices rather than individual drives, you can set GreyHole to zero redundancy and just use it to pool your storage. I'd do this as a temporary fix until e2fsprogs 1.42 is released, or you find a way to back up your data and move to another filesystem.

I too would recommend ZFS on Linux; KQ Infotech look to have done a very good job.

 

only real two options IMHO for >16 TiB support is JFS or XFS. I have really been bitten badly by XFS on mutliple things in the past. I hear its better but JFS has been extremely reliable for me and it should work for file-systems up to 512 TiB.

 

I've been looking into this and it seems resize2fs has been updated to support >16tb.

http://e2fsprogs.sourceforge.net/e2fsprogs-release.html#1.42

http://blog.ronnyegner-consulting.de/2011/08/18/ext4-and-the-16-tb-limit-now-solved/

 

You have to wait for a 3.2.X kernel however.

 

I did a quick Google search after reading this. It seems the 3.2 kernel is in RC7 and they said that's the last one before release. So hopefully once the repos get it, it will be as simple as upgrading all required software and have at it. Or I hope.

 

On my home NAS, Im running CentOS 6.2. I found a guide for Fedora on how to install the 3.2 kernel. I gave it a shot following the directions and it went flawlessly. At least so far.

Code:

[root@Mega-Nas ~]# uname -r
3.2.0

Here is the site that helped me. They also have directions for a Ubuntu based system also.

http://www.howopensource.com/2011/08/how-to-install-compile-linux-kernel-3-0-in-fedora-15-and-14/

 

Did 3.2.0 give you > 16TB ext4 filesystems? I already have 3.2.1 on several of my servers here at work.

 

I don't have enough hard drive space to check. I am mainly doing this in preparation for once I get there.

Another thing I did was update E2fsprogs to the latest 1.42 from directions from this site.


http://blog.ronnyegner-consulting.de/2011/08/18/ext4-and-the-16-tb-limit-now-solved/

Code:

[root@Mega-Nas build]# tune2fs
tune2fs 1.42 (29-Nov-2011)
Usage: tune2fs [-c max_mounts_count] [-e errors_behavior] [-g group]
        [-i interval[d|m|w]] [-j] [-J journal_options] [-l]
        [-m reserved_blocks_percent] [-o [^]mount_options[,...]] [-p mmp_update_interval]
        [-r reserved_blocks_count] [-u user] [-C mount_count] [-L volume_label]
        [-M last_mounted_dir] [-O [^]feature[,...]]
        [-E extended-option[,...]] [-T last_check_time] [-U UUID]
        [ -I new_inode_size ] device

With both of these installed, I think I should be able to create a >16tb file system, just not resize one yet.

So my thought would be once E2fsprogs is updated to allow for resizing above 16tb, I,We should be ready to go, no?

 

Just thought I would mention this. Use caution with this. Its probably something I did, but my raid array wont start on boot any more.

http://hardforum.com/showthread.php?t=1667805

 

Ugh, you're already way out of my comfort zone with the bleeding edge stuff. I'd switch to a FS that supports it better, but that's just me...

 

I agree, the problems you are seeing is an indication that very few people are using ext4 on volumes that large. I'd be very hesitant to use resize2fs for a while after they fix the 16TB limit.

 

had this very same issue, 13x1,5 r6 expanded to 18x w/ no problem, but exty failed on me, thankfully i had 5empty drives, so it took a week but im on jfs now, problem fixed

 

Once E2fsprogs is updated it will be able to resize above 16tb.

http://e2fsprogs.sourceforge.net/e2fsprogs-release.html#1.42

 

It does not look like the update made it into 3.2. Here is what I see from the changes related to ext4:

Source:
http://kernelnewbies.org/LinuxChanges#head-e3ffec0bbdab939b186cd5ec73a4eb40a5a766d0

 

Ok, I finally have enough drives to put me above 16tb. I started researching it some more and found this.

http://www.phoronix.com/scan.php?page=news_item&px=MTA0MDg

So I updated my system, Debian Wheezy, to the 3.3 kernel. Im also running e2fsprogs 1.42.1-2 which should support online resizing.

But when I tried to resize from 12tb to over 16tb, Im getting the famous message.

Code:

root@mega:~# resize2fs /dev/md0
resize2fs 1.42.1 (17-Feb-2012)
resize2fs: New size too large to be expressed in 32 bits

Upon further research, I found this.

http://ubuntuforums.org/showpost.php?p=10995759&postcount=6

So, it seems that if I back up my data, wipe the drives, make a new file system, we should be good to go?

 

I found this thread quite useful so I decided to share my experiences as well.

I'd just like to mention here that I found that ext4 resizing to larger then 16TB did not work with kernel 3.2.0 and e2fsprogs 1.42. either online or offline.
and while this may have been possible with the 3.3.0 kernel I'm not comfortable putting my data that close to beta versions of a kernel.

but btrfs-convert works fine and then a
"btrfs filesystem resize max /mnt"
appears to have done the resizing fine.

so far this works fine, I am leaving the ext4 snapshot in place for a bit though.
(btrfs-convert can do a roll back, though if you use checksums(default) it'll take a long time to do the conversion and that is offline only.
with btrfs on top of (sw) raid6 checksums seemed a bit redundant as well.

just my 2 cents.

I realise btrfs is still a little bit in beta but with the roll-back option i can presumably go back to what I had before, so no data loss except for the new data...

it's like having a snapshot only of the old FS as well as of the old files...
the alternative (for me)was getting a couple of drives out of the raid setup (re-syncing for each drive) and then running user-space ZFS on the freed drives and copying chunks there and grow the zfs as it goes on etc, I chose to be lazy...
of course my limitations may be that I don't have a place to back up 10TB of data etc... if I'd have that option I would take it and run on ZFS from then on..

the net result is i have my bigger FS and still have all my data without all the trouble it would have taken to get that.

 

I would be careful to trust btrfs in some years. Last month, it corrupts and loose all data, new and old:
http://phoronix.com/forums/showthre...ion-Ready-In-Oracle-Linux&p=249731#post249731

"...After this system was completely corrupted and it is not recoverable.
....
I think Oracle need to spend couple more years before [BTRFS] will work...."

 

BTW, ext4 is now supposed to be able to online resize > 16TB with kernel 3.7.

http://www.h-online.com/open/features/What-s-new-in-Linux-3-7-1759862.html