![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
RaidZ3 has much smaller available disk space than expected
- Thread starter MarcusXP
- Start date
I don't believe this problem is related to the number of drives in the pool, it seems to be simply because of ASHIFT value.
The 512b sector size was very good because probably the filesystem's metadata fit well in that size.
The 4K sector size is probably way too big for the metadata so much of that space is wasted.
-- At least that is my understanding, someone correct me if I am wrong
Last edited:
Can you please explain how to force ASHIFT=9 on OpenIndiana?
I don't care as much about the performance, but I do care about the extra space wasted in this process...
Yes, using 4k for metadata, that is normally compressed and fits well into 512b is a bit of a waste, but it's not normally a huge amount of waste.
Normally the huge amount of waste is when you use a raidz, with a odd number of disks (number of data disks does not end up in evenly divisable by 4k, when 128k / data-disk-count). So this means you will waste space, and normally quickly, when storing files, on it. Also the freespace calculation takes assumes 128k or larger files will be stored on it, so it will report freespace based on this lost space.
If your storing smaller files, raidz will switch to using mirrors on the raidz disks to store files, causing even more space to be used (but less than if it attempted to stripe that file over the 23 disks)
This affects 4k and 512b drives the same. The issue is, the most amount of waste you can have is 511b on those disks, so it doesn't account for a lot of waste, compared to 4k where you normally wasting a good 1-2k per disk.
Normally the huge amount of waste is when you use a raidz, with a odd number of disks (number of data disks does not end up in evenly divisable by 4k, when 128k / data-disk-count). So this means you will waste space, and normally quickly, when storing files, on it. Also the freespace calculation takes assumes 128k or larger files will be stored on it, so it will report freespace based on this lost space.
If your storing smaller files, raidz will switch to using mirrors on the raidz disks to store files, causing even more space to be used (but less than if it attempted to stripe that file over the 23 disks)
This affects 4k and 512b drives the same. The issue is, the most amount of waste you can have is 511b on those disks, so it doesn't account for a lot of waste, compared to 4k where you normally wasting a good 1-2k per disk.
glassdarkly
n00b
- Joined
- Oct 21, 2009
- Messages
- 5
Silhouette said:
Did you happen to compare speeds? I've read that the performance impact of using ashift = 9 on 4k disks can be significant, especially with resilver times.
To summarize, it seems that there are at least two things that contribute to waste of disk space on ZFS (but there may be some other factors as well):
- disk sector size (4k vs 512b)
- number of data disks in a vdev (if it's not a number 2^n, more disk space is wasted).
This should have been documented somewhere.. but I haven't heard of this until I actually set-up a ZFS box and had all these issues... duh!
I'm gonna try to set-up a pool with 4 x VDEVS of 6 drives each RAIDZ2 (4 data + 2 parity on each VDEV). This should work pretty well I guess.
The problem is my system (not sure exactly which component) only sees 23 disks through the integrated SAS expander on the Supermicro case... although 24 disks are plugged
I'm doing more investigation to see how I can get this solved.. trying other raid controllers (HBAs) and different motherboards.. hopefully I will get it sorted soon(ish).
- disk sector size (4k vs 512b)
- number of data disks in a vdev (if it's not a number 2^n, more disk space is wasted).
This should have been documented somewhere.. but I haven't heard of this until I actually set-up a ZFS box and had all these issues... duh!
I'm gonna try to set-up a pool with 4 x VDEVS of 6 drives each RAIDZ2 (4 data + 2 parity on each VDEV). This should work pretty well I guess.
The problem is my system (not sure exactly which component) only sees 23 disks through the integrated SAS expander on the Supermicro case... although 24 disks are plugged
I'm doing more investigation to see how I can get this solved.. trying other raid controllers (HBAs) and different motherboards.. hopefully I will get it sorted soon(ish).
Silhouette
Limp Gawd
- Joined
- Dec 14, 2006
- Messages
- 205
I did thorough testing of the servers before I decided on what I should do and started using them. The performance is noticeably lower, both while doing resilvers and transferring files over 10GbE network. For those servers I only store relatively small files anyway. My resilver speed is still acceptable to me, but this needs proper testing on each system with a pool that contains a reasonable amount of file data. The two servers that I forced ashift=9 on had a relatively different performance drop.
Ufortunately I don't know if there is a way to force the lower ashift on openindiana.
The problem is my system (not sure exactly which component) only sees 23 disks through the integrated SAS expander on the Supermicro case... although 24 disks are plugged
I'm doing more investigation to see how I can get this solved.. trying other raid controllers (HBAs) and different motherboards.. hopefully I will get it sorted soon(ish).
How are you trying to FIND the drives?
eg using the format command? or thru a napp-it type interface?
Which OS are you using etc?
I had some troubles with napp-it finding the correct drives.... format command found the missing drives... but was a bit messy in its output of the drives
As your still in a testing faze.....
Can you try Solaris 11 out (Not SolarisExpress 11)
Then use the croinfo command to give us a looksee at what the backplane reports it has connected
hopefully your backplane has a better SES2 standard implementation than mine
A Quick guide here on how to use it.
http://forums.overclockers.com.au/showpost.php?p=14424399&postcount=752
Trust the people that are telling you that's too many drives in one VDEV. While it will work, when you experience a failure, it's going to suck big time. I recently had a very stressful 2-3 weeks because I had 24 drives in a VDEV, and I will never do it again. Also, the performance is significantly better when split into more than one VDEV. I'd recommend 3 8 drive VDEVs.
No, it's some kind of firmware issue.
After many hours of testing, the problem went away after I changed the motherboard.
I am now using a Tyan dual LGA1366 motherboard (I was using Asus dual LGA1366 motherboard before), and I can see all 24 drives (plus the one drive for OS).
The weird part is, I even tried with another controller (Areca 1680IX-24 which I changed to JBOD mode --- kind of like IT mode for LSI) and on the Areca, only 23 drives would show up as well, not 24.
And it seems that it was random, sometimes another drive was missing, not same drive every time.
Looked like some kind of resource conflict, but I didn't have the patience troubleshooting it further. Plus, I like the Tyan board better
I am using OpenIndiana, and I checked the drives in Napp-it and in the SSH (running "format" utility, it would only list 23 drives + the OS drive, not 24 drives + the OS drive, as expected).
Last edited:
I am using OpenIndiana.. and "format" utility doesn't see all drives either. It lists same drives as Napp-It...
Eventually the problem went away when I changed the motherboard. Weird, I know... but it drove me crazy for a few days
I ended up with the following config:
Pool1: 1 VDEV of 19 drives (16 + 3) RaidZ3 -- About 54TB usable space
Pool2: 1 VDEV of 5 drives (3 + 2) RaidZ2 --- About 10.2TB usable space
Pool 2 not optimal, optimal would have been 4 +1 drives RaidZ, but I wanted better redundancy. So I sacrificed some disk space and performance for it.
I created 1 VDEV on each Pool so I can separate the 5 drive VDEV which has weaker redundancy.
So I have now 2 volumes (one on each pool), assigned to 1 FibreChannel port.. working pretty nice.
I get about 360-370MB/sec read and 120-160MB/sec write speed through FiberChannel on each volume, tested with Windows and Linux clients (virtual machines which have the FC card physically assigned to them in ESXi).
I did try the following config:
4 VDEVS of 6 Drives each (4 + 2) RaidZ2, in one Pool, and the performance results 360-370MB/sec read and 250-260MB/sec write.
I loose some write speed, but gained over 10TB more of disk space... while still having very good redundancy.
I am pretty happy so far.. I will set-up a 2nd similar storage server in the next week or two I hope
Pool1: 1 VDEV of 19 drives (16 + 3) RaidZ3 -- About 54TB usable space
Pool2: 1 VDEV of 5 drives (3 + 2) RaidZ2 --- About 10.2TB usable space
Pool 2 not optimal, optimal would have been 4 +1 drives RaidZ, but I wanted better redundancy. So I sacrificed some disk space and performance for it.
I created 1 VDEV on each Pool so I can separate the 5 drive VDEV which has weaker redundancy.
So I have now 2 volumes (one on each pool), assigned to 1 FibreChannel port.. working pretty nice.
I get about 360-370MB/sec read and 120-160MB/sec write speed through FiberChannel on each volume, tested with Windows and Linux clients (virtual machines which have the FC card physically assigned to them in ESXi).
I did try the following config:
4 VDEVS of 6 Drives each (4 + 2) RaidZ2, in one Pool, and the performance results 360-370MB/sec read and 250-260MB/sec write.
I loose some write speed, but gained over 10TB more of disk space... while still having very good redundancy.
I am pretty happy so far.. I will set-up a 2nd similar storage server in the next week or two I hope