Switch to ZFS and losing space?

O

OldSchool

Limp Gawd
Joined
Jul 6, 2010
Messages
477
Over the past week I have upgraded my storage server and transitioned from a hardware RAID 6 on an Areca arc-1880i to zfs on freebsd. I am using encryption as well, the hardware raid was encrypted in Truecrypt and the card was assigned to a Windows Server 2K8 R2 guest, now the drives are connected to an HBA and an HP SAS exapnder that is passed through to a freebsd guest and all of the drives are encrypted with GELI.

Basically the process went like this:

Current 8x2TB Hitachi 7K3000s RAID 6 on ARC-1880i

Drives encrypted with GELI, new Raidz2 of 8x3TB WD Reds created on freebsd

Data from hardware RAID 6 copied to new 8x3TB Raidz2

Hardware raid card removed and SAS expander installed with all 16 drives attached and dual linked to HBA

2nd Raidz2 created with 8x2TB Hitachi drives encrypted with GELI

Data that was stored on hardware RAID 6 copied from 8x3TB back to 8x2TB


Everything seemed to be going fine, until I noticed something... The data that was on the hardware RAID 6 was 5.54TB before being temporarily moved to the 8x3TB raidz2. When it was copied back to the newly created 8x2TB raidz2 it now takes up 6.72TB!?!

Wtf? Any ideas?
 
did you have compression enabled in windows?

i'm not positive what the windows compression defaults too but it is likely that it is slightly more aggressive than the default lzjb in zfs.

the more likely difference may be block size though differences though. ntfs defaults to 4K allocation size for normal size volumes iirc once you get above 13TB that default jumps to 8K. what is the dataset size in your ZFS?
 
All of the drives were encrypted in GELI w/ 4K sectors, which I believe is the default. I tried using 512 with the Hitachis because I believe they emulate 512, but the performance of the array ended up being worse. Compression wasn't enabled in windows, and isn't enabled with zfs. The data isn't really compressible.
 
Something that nobody really talks about when it comes to raid is stripe size. One thing I noticed when I toyed round with the stripe size on my areca raid arrays is not only does it affect performance but it also affects hard drive space. I figured out that the higher the stripe size on the array the more hard drive space files took up. What stripe size is each array using?

edit: filesystem cluster size impacts this as well.
 
LBJM said:
Something that nobody really talks about when it comes to raid is stripe size. One thing I noticed when I toyed round with the stripe size on my areca raid arrays is not only does it affect performance but it also affects hard drive space. I figured out that the higher the stripe size on the array the more hard drive space files took up.
Click to expand...

Huh? How did you come to that confusion? :) Size of raid blocks and on-disk format (filesystem layer) are totally separate, nothing to do with one another, the raid controller has no filesystem awareness - its job is creating a virtual harddisk by grouping a set of physical disks together and interleaving them in a pattern of blocks (stripesize = size of those blocks) to present what appears as one big harddisk to the host O/S. Conversely, the filesystem layer doesn't have any awareness of that interleaving mechanism its sitting on top of - all it sees is one big "harddisk" with a start sector and end sector.
 
Last edited:
Does the original source have any entries with hardlink count greater than 1?
 
GELI is like TrueCrypt a transparaent block-based encryption layer. If you encrypt a 1TB device, you will get 1TB sized block device on top of that, minus a few kilobytes for headers/bootloaders. The blocksize only influences performance.

Does your filesystem contain a large amount of small files? I believe NTFS packs multiple small files into a single cluster, while ZFS does not.
 
OldSchool said:
Over the past week I have upgraded my storage server and transitioned from a hardware RAID 6 on an Areca arc-1880i to zfs on freebsd. I am using encryption as well, the hardware raid was encrypted in Truecrypt and the card was assigned to a Windows Server 2K8 R2 guest, now the drives are connected to an HBA and an HP SAS exapnder that is passed through to a freebsd guest and all of the drives are encrypted with GELI.

Basically the process went like this:

Current 8x2TB Hitachi 7K3000s RAID 6 on ARC-1880i

Drives encrypted with GELI, new Raidz2 of 8x3TB WD Reds created on freebsd

Data from hardware RAID 6 copied to new 8x3TB Raidz2

Hardware raid card removed and SAS expander installed with all 16 drives attached and dual linked to HBA

2nd Raidz2 created with 8x2TB Hitachi drives encrypted with GELI

Data that was stored on hardware RAID 6 copied from 8x3TB back to 8x2TB


Everything seemed to be going fine, until I noticed something... The data that was on the hardware RAID 6 was 5.54TB before being temporarily moved to the 8x3TB raidz2. When it was copied back to the newly created 8x2TB raidz2 it now takes up 6.72TB!?!

Wtf? Any ideas?
Click to expand...

The answer is that you are looking at your ZFS Pool, and not the ZFS folder.

The ZFS Pool lists data size as seen on drives (so includes redundancy), while the ZFS folder lists the size, as seen on the filesystem.

For example, on my "pool" fs6 I got Alloc 34.1T, but on my "folder" I got Used 26.0T.

Makes sense ?
 
You must log in or register to reply here.
Back
Top