• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Resizing ext4 filesystem > 16TB

A

apekatt

n00b
Joined
Jul 31, 2009
Messages
16
I have file server that I've kept adding disks to for many years now, and I recently hit a wall when I reached 16TB. The problem from what I can find out is that resize2fs cant resize filesystems past 16TB using ext4. I've been googling a bit and it seems the latest version of resize2fs from 07-02-11 still doesn't have support for this, and it is a WIP ( See screenshot ).

Now I'm wondering what my options are.

1. Find a workaround so I can resize me existing filesystem. ( Wait for an update to resize2fs. )
2. Convert to another filesystem.
3. Make another LVM Volume.
4. Reformat the filesystem with another filesystem.

I would prefer to keep the system as one big volume, and not split it in two.
Alternatives 1 or 2 would be preferable.

ext4_16tb.png
 
I think the best option would be to look at other filesystems such as xfs or zfs... Unfortunately that means backing up everything and reformatting.

e2fsprogs still does not support >16TB officially and any of the beta releases are buggy as hell. Not something I would trust with my data. It's been a known issue for years so I wouldn't wait for them to release a stable version. ;)
 
Might want to take a look at zfs. Supposedly the zfs on linux project is stable now. If not, you could maybe move to OpenIndiana or SolarisExpress11 with zfs.
 
Anyway you look at it, at present you're going to need to look at option 4 at the moment unless you have the ability to wait until they have the issues with EXT4 sorted out and proven.

If you can't I'd recommend JFS as a suitable filesystem as its often overlooked down to it not being actively developed by IBM. The reason however for that is different from the norm however, as from what I understand, it is that they don't need to add any further features but are still very much in the development game with it.

Houkouonchi uses it on this very large arrays with no issues and I also do on my (rather small) 18tb RAID-6 array and I've never had any problems with it so far. A few others also do, the most read about example being Backblaze on their Petabyte storage platform.
 
Yeah, I thought about converting to Btrfs, but not sure if it's stable enough. Read some reports of people loosing data after power failures etc.

Maybe I'll just wait and see what comes first; Btrfs fschk or updated version of e2fsprogs.
 
Another option would to create another LVM and then pool the two volumes using GreyHole. This will present a single share over the network, but obviously they will still be two separate volumes on the server itself.

Since (I assume) your PVs are in fact RAID devices rather than individual drives, you can set GreyHole to zero redundancy and just use it to pool your storage. I'd do this as a temporary fix until e2fsprogs 1.42 is released, or you find a way to back up your data and move to another filesystem.

I too would recommend ZFS on Linux; KQ Infotech look to have done a very good job.
 
only real two options IMHO for >16 TiB support is JFS or XFS. I have really been bitten badly by XFS on mutliple things in the past. I hear its better but JFS has been extremely reliable for me and it should work for file-systems up to 512 TiB.
 
You have to wait for a 3.2.X kernel however.

This release of e2fsprogs has support for file systems > 16TB. Online resize requires kernel support which will hopefully be in Linux version 3.2. Offline support is not yet available for > 16TB file systems, but will be coming.
Click to expand...
 
drescherjm said:
You have to wait for a 3.2.X kernel however.
Click to expand...

I did a quick Google search after reading this. It seems the 3.2 kernel is in RC7 and they said that's the last one before release. So hopefully once the repos get it, it will be as simple as upgrading all required software and have at it. Or I hope.
 
Did 3.2.0 give you > 16TB ext4 filesystems? I already have 3.2.1 on several of my servers here at work.
 
drescherjm said:
Did 3.2.0 give you > 16TB ext4 filesystems? I already have 3.2.1 on several of my servers here at work.
Click to expand...

I don't have enough hard drive space to check. I am mainly doing this in preparation for once I get there.

Another thing I did was update E2fsprogs to the latest 1.42 from directions from this site.


http://blog.ronnyegner-consulting.de/2011/08/18/ext4-and-the-16-tb-limit-now-solved/

Code: 
[root@Mega-Nas build]# tune2fs
tune2fs 1.42 (29-Nov-2011)
Usage: tune2fs [-c max_mounts_count] [-e errors_behavior] [-g group]
        [-i interval[d|m|w]] [-j] [-J journal_options] [-l]
        [-m reserved_blocks_percent] [-o [^]mount_options[,...]] [-p mmp_update_interval]
        [-r reserved_blocks_count] [-u user] [-C mount_count] [-L volume_label]
        [-M last_mounted_dir] [-O [^]feature[,...]]
        [-E extended-option[,...]] [-T last_check_time] [-U UUID]
        [ -I new_inode_size ] device

With both of these installed, I think I should be able to create a >16tb file system, just not resize one yet.

So my thought would be once E2fsprogs is updated to allow for resizing above 16tb, I,We should be ready to go, no?
 
Just to update. I have been testing a bit with new kernel, and I still cant resize it past 16TB.

I can create a new file system bigger than 16tb, but once I try to resize it again it fails.

This is my test system in Vmware. Just added a bunch of disks with thin provisioning to test:

ext4_16tb_2.png


Any suggestions? Can I do something with tune2fs to make it work?
 
Ugh, you're already way out of my comfort zone with the bleeding edge stuff. I'd switch to a FS that supports it better, but that's just me...
 
I agree, the problems you are seeing is an indication that very few people are using ext4 on volumes that large. I'd be very hesitant to use resize2fs for a while after they fix the 16TB limit.
 
had this very same issue, 13x1,5 r6 expanded to 18x w/ no problem, but exty failed on me, thankfully i had 5empty drives, so it took a week but im on jfs now, problem fixed :D
 
apekatt said:
Just to update. I have been testing a bit with new kernel, and I still cant resize it past 16TB.
Click to expand...
Once E2fsprogs is updated it will be able to resize above 16tb.

This release of e2fsprogs has support for file systems > 16TB. Online resize requires kernel support which will hopefully be in Linux version 3.2. Offline support is not yet available for > 16TB file systems, but will be coming.
Click to expand...

http://e2fsprogs.sourceforge.net/e2fsprogs-release.html#1.42
 
It does not look like the update made it into 3.2. Here is what I see from the changes related to ext4:

1.1. ext4: Support for bigger block sizes

Recommended LWN article: Improving ext4: bigalloc, inline data, and metadata checksums

The maximum size of a filesystem block in ext4 has always been 4 KB in x86 systems. But the storage capacity of modern hard disks is growing fast, and with the size of hard disks, the overhead of using such small size as block size increases. Small block sizes benefit users who have many small files, because the space will be used more efficiently, but people who uses large files would benefit of larger block sizes.

ext4 supports now block sizes of up to 1MB of size, which decreases considerably the time spent doing block allocations, and there is smaller fragmentation. These new block sizes must be set at creation time, using the mkfs -C option (requires e2fsprogs 1.42). This feature is not backwards compatible with older kernels. Code: (commit 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17)
Click to expand...

Source:
http://kernelnewbies.org/LinuxChanges#head-e3ffec0bbdab939b186cd5ec73a4eb40a5a766d0
 
Ok, I finally have enough drives to put me above 16tb. I started researching it some more and found this.

http://www.phoronix.com/scan.php?page=news_item&px=MTA0MDg

The EXT4 file-system pull request went in today for the Linux 3.3 kernel. The EXT4 updates for this next kernel include the new online resize interface.

Among various bug-fixes and other changes to the current de facto standard Linux file-system, EXT4 now has the new online resize interface, which was first publicly introduced a few months back. The new online resize interface for EXT4 is the work of Yongqiang Yang.

As far as what the new online resize interface is for with EXT4, "It is a new online resize interface for ext4. It can be used via ioctl with EXT4_IOC_RESIZE_FS and a 64 bit integer indicating size of the resized fs in block."

What makes Yongqiang's implementation different though from the status quo online resize of EXT4 is that the kernel now does all the work. The resizing now is also much faster. Benchmarks shown by Yongqiang indicate the file-system resize operation with this new code is now significantly faster. A 20GB resize on the old code toook over five minutes with the pre-3.3 implementation, but now he's shown that it can be done in just 3.35 seconds. A 100GB resize can be done in less than a half second.

Besides being much faster for resizing the file-system, the new code can also more easily handle new features. "New resize can support new feature like bigalloc and exclude bitmap easily. Because it lets kernel do all work."

The EXT4 new online resize code is specifically talked about in this mailing list message (among others) from the past few months. The new resize code adds in around 1,000 lines of code to the file-system's kernel module.

The pull request for the Linux 3.3 kernel that includes this work plus other EXT4 changes can be found in this email from Ted Ts'o.

This EXT4 improvement is just one of the many features of the Linux 3.3 kernel, which has just begun development and will be officially released in about two months. The Btrfs file-system pull request for 3.3 has yet to hit, but we'll see if it brings work like the pretty beefy changes it had in the last kernel cycle. Hopefully you're also not waiting on the Reiser4 file-system to go mainline.
Click to expand...

So I updated my system, Debian Wheezy, to the 3.3 kernel. Im also running e2fsprogs 1.42.1-2 which should support online resizing.

But when I tried to resize from 12tb to over 16tb, Im getting the famous message.

Code: 
root@mega:~# resize2fs /dev/md0
resize2fs 1.42.1 (17-Feb-2012)
resize2fs: New size too large to be expressed in 32 bits

Upon further research, I found this.

http://ubuntuforums.org/showpost.php?p=10995759&postcount=6

There isn't a way to get around this issue, I'm afraid. Support for >
16TB file systems is something that requires a fundamental change in
the file system format, and which is in late beta status.

It requires the pre-release, e2fsprogs 1.42 development branch, and
unfortunately, the file system must be reformatted to support larger
(greater than 32-bit) block numbers. Certain on-disk data structures
(specifically, the block group descriptors) have to grow in size from
32 bytes to 64 bytes, so it's not something where we could upgrade a
file system in-place to support > 16TB.

Sorry, I'm afraid can't give you a magic solution to your problem....

Regards,

- Ted
Click to expand...

So, it seems that if I back up my data, wipe the drives, make a new file system, we should be good to go?
 
I found this thread quite useful so I decided to share my experiences as well.

I'd just like to mention here that I found that ext4 resizing to larger then 16TB did not work with kernel 3.2.0 and e2fsprogs 1.42. either online or offline.
and while this may have been possible with the 3.3.0 kernel I'm not comfortable putting my data that close to beta versions of a kernel.

but btrfs-convert works fine and then a
"btrfs filesystem resize max /mnt"
appears to have done the resizing fine.

so far this works fine, I am leaving the ext4 snapshot in place for a bit though.
(btrfs-convert can do a roll back, though if you use checksums(default) it'll take a long time to do the conversion and that is offline only.
with btrfs on top of (sw) raid6 checksums seemed a bit redundant as well.

just my 2 cents.

I realise btrfs is still a little bit in beta but with the roll-back option i can presumably go back to what I had before, so no data loss except for the new data...

it's like having a snapshot only of the old FS as well as of the old files...
the alternative (for me)was getting a couple of drives out of the raid setup (re-syncing for each drive) and then running user-space ZFS on the freed drives and copying chunks there and grow the zfs as it goes on etc, I chose to be lazy...
of course my limitations may be that I don't have a place to back up 10TB of data etc... if I'd have that option I would take it and run on ZFS from then on..

the net result is i have my bigger FS and still have all my data without all the trouble it would have taken to get that.
 
Last edited:
robinmdh said:
I realise btrfs is still a little bit in beta but with the roll-back option i can presumably go back to what I had before, so no data loss except for the new data...
Click to expand...
I would be careful to trust btrfs in some years. Last month, it corrupts and loose all data, new and old:
http://phoronix.com/forums/showthre...ion-Ready-In-Oracle-Linux&p=249731#post249731

"...After this system was completely corrupted and it is not recoverable.
....
I think Oracle need to spend couple more years before [BTRFS] will work...."
 
You must log in or register to reply here.
Back
Top