![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
Testing ZFS RAID-Z performance with 4K sector drives
- Thread starter sub.mesa
- Start date
hello all. I have been trying to understand zfs for about a week or more now and building a server and the abundance of information is a bit overwhelming. I am still trying to get my head wrap around using ESXi and having multiple VMs. My two questions (I have many more but these two apply to this thread) is
1. A zPool is created with a single vdev (which can be a single disk or multiple disks), or multiple vdevs. Correct? Are these benchmarks assuming one vdev (I.E. one vdev consisting of 4,5,6,7...disks)?
2. I am a bit confused with the magic number of disks and this relates to my first question to (magig number in vdevs assuming multiple vdevs or the magical number in a pool period). I plan on building a server in the coming months (which should give me a bit more time to identify what I need/want). I read earlier, sub.mesa that data, optimally should be written in 32-128KB blocks, but yet, going to alot more disks offers better performance. I feel like the optimal solution for writing to a disk is to decrease it (going from 32KB->16KB) as it doesn't need to spend that much time writing to the disk. As long as it can go into 128KB evenly?
Thanks for your help. I having other questions about zfs and using virtual machines, and this and that hardware but I am having a hard time finding the correct thread to read up on it/post a question. sub.mesa, did a good writeup on zfs in his zfs.guru thread. Thank you for that. Does anyone know a good thread that talks about using esxi and vti-d, etc etc. I know the saying that if you don't know what it is, you probably don't need it, but I am quite sure many of us didn't pop out of the womb knowing what it was either (and for some, didn't know what it was because it didn't exist yet)
. I would like to learn about it and identify if I can take advantage of it or not. Thanks again.
1. A zPool is created with a single vdev (which can be a single disk or multiple disks), or multiple vdevs. Correct? Are these benchmarks assuming one vdev (I.E. one vdev consisting of 4,5,6,7...disks)?
2. I am a bit confused with the magic number of disks and this relates to my first question to (magig number in vdevs assuming multiple vdevs or the magical number in a pool period). I plan on building a server in the coming months (which should give me a bit more time to identify what I need/want). I read earlier, sub.mesa that data, optimally should be written in 32-128KB blocks, but yet, going to alot more disks offers better performance. I feel like the optimal solution for writing to a disk is to decrease it (going from 32KB->16KB) as it doesn't need to spend that much time writing to the disk. As long as it can go into 128KB evenly?
Thanks for your help. I having other questions about zfs and using virtual machines, and this and that hardware but I am having a hard time finding the correct thread to read up on it/post a question. sub.mesa, did a good writeup on zfs in his zfs.guru thread. Thank you for that. Does anyone know a good thread that talks about using esxi and vti-d, etc etc. I know the saying that if you don't know what it is, you probably don't need it, but I am quite sure many of us didn't pop out of the womb knowing what it was either (and for some, didn't know what it was because it didn't exist yet)
. I would like to learn about it and identify if I can take advantage of it or not. Thanks again.
Last edited:
Flintstone
Limp Gawd
- Joined
- Nov 12, 2010
- Messages
- 130
Just to be clear. ZFS will detect bad cables, bad controllers, bad memory, bad PSU, bad motherboard and bad hotplug bays. Of course issues can occure "outside" the filesystem and not be detected (like network traffic not beeing stored), but if you get errors on your ZFS system (atleast when more than one drive is involved) you better start checking your hardware
I've detected bad cables (several), bad harddrives, bad HBA drivers (old Sil that wasn't supported in Solaris10), a bad hotplug bay and now bad harddrive firmware - all because of ZFS.
Last edited:
So i just wanted to clarify some things to myself. The hardware configuration for a server i'm currently building is as follows:
SuperMicro X8DTE-F
Intel Xeon E5530
12GB DDR3
3Ware 9650SE-16L
3x Hitachi 500GB raid sata-disks (HW RAID1 + hot spare) (for base system, running UFS)
My system is currently running FreeBSD 8.2 stable with zfs v15. I started planning on buying some data disks too, and before finding all this 4K sector stuff I ordered myself 7 WD20EARS drives. Now i planned on using them as 6 drives in one RAID-Z array + 1 hot spare. But now i keep reading here, that 6 drive RAID-Z configuration is not optimal? It would however fit so nicely for my remaining 13 free hotplug slots, since i plan to expand later on with other 6 drive RAID-Z array and keeping that one hot spare. Should i go with 6 drive RAID-Z or try something else?
Also i'm very confused about this glob 4k sector thing. Can someone confirm if i need to patch my sources to make it work, or will it simply be sufficient to use it prior to zfs create? And will the settings stick?
sub.mesa, what is your take on NCQ on those WD drives? They should be new ones. I don't even need very high transfer rates, since my server is basically for shell use. I would simply love steady operation and decent speeds.
Then final note, has anyone experimented with wdidle.exe to disable WD power saving features, like head parking? Or is this even an issue with ZFS?
Sorry for being a little messy but this is my first post in a long time
SuperMicro X8DTE-F
Intel Xeon E5530
12GB DDR3
3Ware 9650SE-16L
3x Hitachi 500GB raid sata-disks (HW RAID1 + hot spare) (for base system, running UFS)
My system is currently running FreeBSD 8.2 stable with zfs v15. I started planning on buying some data disks too, and before finding all this 4K sector stuff I ordered myself 7 WD20EARS drives. Now i planned on using them as 6 drives in one RAID-Z array + 1 hot spare. But now i keep reading here, that 6 drive RAID-Z configuration is not optimal? It would however fit so nicely for my remaining 13 free hotplug slots, since i plan to expand later on with other 6 drive RAID-Z array and keeping that one hot spare. Should i go with 6 drive RAID-Z or try something else?
Also i'm very confused about this glob 4k sector thing. Can someone confirm if i need to patch my sources to make it work, or will it simply be sufficient to use it prior to zfs create? And will the settings stick?
sub.mesa, what is your take on NCQ on those WD drives? They should be new ones. I don't even need very high transfer rates, since my server is basically for shell use. I would simply love steady operation and decent speeds.
Then final note, has anyone experimented with wdidle.exe to disable WD power saving features, like head parking? Or is this even an issue with ZFS?
Sorry for being a little messy but this is my first post in a long time
I figured, enough with benchmarking and it's about time to get an everyday feel of the server.
Created a raidz1 pool (5 x 2TB western digital) with 4k override. So far server is storing around 2.7TB of data. If I got to say, I'm pleased with the performance: samba, iscsi, non-destrucitve benchmark. Might as well move it to production use.
So did WD improve their recent HDD's performance, or is it the magic effect of mesa's ashift trick?
The only caveat after using 4k-override is that HDDs are referenced by partition into the pool instead of the gpt label. Not a real issue. Just wondering what's causing it.
Created a raidz1 pool (5 x 2TB western digital) with 4k override. So far server is storing around 2.7TB of data. If I got to say, I'm pleased with the performance: samba, iscsi, non-destrucitve benchmark. Might as well move it to production use.
So did WD improve their recent HDD's performance, or is it the magic effect of mesa's ashift trick?
The only caveat after using 4k-override is that HDDs are referenced by partition into the pool instead of the gpt label. Not a real issue. Just wondering what's causing it.
Indiana-Joe
n00b
- Joined
- Jan 4, 2011
- Messages
- 13
[EDIT] post removed
Last edited:
hello all. I have been trying to understand zfs for about a week or more now and building a server and the abundance of information is a bit overwhelming. I am still trying to get my head wrap around using ESXi and having multiple VMs. My two questions (I have many more but these two apply to this thread) is
1. A zPool is created with a single vdev (which can be a single disk or multiple disks), or multiple vdevs. Correct? Are these benchmarks assuming one vdev (I.E. one vdev consisting of 4,5,6,7...disks)?
2. I am a bit confused with the magic number of disks and this relates to my first question to (magig number in vdevs assuming multiple vdevs or the magical number in a pool period). I plan on building a server in the coming months (which should give me a bit more time to identify what I need/want). I read earlier, sub.mesa that data, optimally should be written in 32-128KB blocks, but yet, going to alot more disks offers better performance. I feel like the optimal solution for writing to a disk is to decrease it (going from 32KB->16KB) as it doesn't need to spend that much time writing to the disk. As long as it can go into 128KB evenly?
Thanks for your help. I having other questions about zfs and using virtual machines, and this and that hardware but I am having a hard time finding the correct thread to read up on it/post a question. sub.mesa, did a good writeup on zfs in his zfs.guru thread. Thank you for that. Does anyone know a good thread that talks about using esxi and vti-d, etc etc. I know the saying that if you don't know what it is, you probably don't need it, but I am quite sure many of us didn't pop out of the womb knowing what it was either (and for some, didn't know what it was because it didn't exist yet)
So i just wanted to clarify some things to myself. The hardware configuration for a server i'm currently building is as follows:
SuperMicro X8DTE-F
Intel Xeon E5530
12GB DDR3
3Ware 9650SE-16L
3x Hitachi 500GB raid sata-disks (HW RAID1 + hot spare) (for base system, running UFS)
My system is currently running FreeBSD 8.2 stable with zfs v15. I started planning on buying some data disks too, and before finding all this 4K sector stuff I ordered myself 7 WD20EARS drives. Now i planned on using them as 6 drives in one RAID-Z array + 1 hot spare. But now i keep reading here, that 6 drive RAID-Z configuration is not optimal? It would however fit so nicely for my remaining 13 free hotplug slots, since i plan to expand later on with other 6 drive RAID-Z array and keeping that one hot spare. Should i go with 6 drive RAID-Z or try something else?
Also i'm very confused about this glob 4k sector thing. Can someone confirm if i need to patch my sources to make it work, or will it simply be sufficient to use it prior to zfs create? And will the settings stick?
sub.mesa, what is your take on NCQ on those WD drives? They should be new ones. I don't even need very high transfer rates, since my server is basically for shell use. I would simply love steady operation and decent speeds.
Then final note, has anyone experimented with wdidle.exe to disable WD power saving features, like head parking? Or is this even an issue with ZFS?
Sorry for being a little messy but this is my first post in a long time
Did you ask this somewhere else? this thread has been a little inactive and wondering if you got your question(s) answered.
I have a mix of 3 Seagate LP drives and 3 WD Green drives. I've had no problems with the combination. I've done a bit of benchmarking, but nothing too fancy.
3 WD Green EARS 2 TB drives and 3 Seagate LP (ST32000542AS) drives
dd write 20 GB @ 247 MBs
dd read 20 GB @ 305 MBs
degraded array (raidz2 6 disks, 5 present -110 GB of each disk allocated, ie 660GB total, 440 GB after parity) - smallest disk was a missing 110 GB vmdk.
dd write 20 GB @ 281 MBs
dd read 20 GB @ 305 MBs
Apparently I never dd benched the Seagates in a raidz
Below are the three drive arrangements (3+3, degraded and 3 Seagates) benchmarked using bonnie:
It seems that speed went up nicely from a 3 disk raidz, to a 6 disc degraded raidz2, and finally the non-degraded 6 disc raidz2 with half Seagate LP drives and half WD greens.
I never did get a chance to run sub.mesa's benchmarks, so these numbers may not be comparable to most in this thread.
3 WD Green EARS 2 TB drives and 3 Seagate LP (ST32000542AS) drives
dd write 20 GB @ 247 MBs
dd read 20 GB @ 305 MBs
degraded array (raidz2 6 disks, 5 present -110 GB of each disk allocated, ie 660GB total, 440 GB after parity) - smallest disk was a missing 110 GB vmdk.
dd write 20 GB @ 281 MBs
dd read 20 GB @ 305 MBs
Apparently I never dd benched the Seagates in a raidz
Below are the three drive arrangements (3+3, degraded and 3 Seagates) benchmarked using bonnie:
It seems that speed went up nicely from a 3 disk raidz, to a 6 disc degraded raidz2, and finally the non-degraded 6 disc raidz2 with half Seagate LP drives and half WD greens.
I never did get a chance to run sub.mesa's benchmarks, so these numbers may not be comparable to most in this thread.
Last edited:
![Zarathustra[H]](https://cdn.hardforum.com/data/avatars/m/9/9298.jpg?1707758471){kind=avatar}
Zarathustra[H]
Extremely [H]
- Joined
- Oct 29, 2000
- Messages
- 38,843
Wow,
So I just read through all this, and I am totally lost.
I've learned that.
1.) Some WD Drives use 4k sector sizes (and emulate 512byte sectors?), and this can be bad for RAIDz performance in some situations.
2.) If you have 4k sector drives, the correct drive quantity is very important?
I recently decided to build a 4 disk RAIDz2 array using 4 WD30EZRX drives.
Doing some research I have found that they are 4k drives like discussed here.
Based on this, am I OK with 4 drives in my array, or should I add a 5th for performance, even though I don't really need the space?
Much appreciated,
Matt
So I just read through all this, and I am totally lost.
I've learned that.
1.) Some WD Drives use 4k sector sizes (and emulate 512byte sectors?), and this can be bad for RAIDz performance in some situations.
2.) If you have 4k sector drives, the correct drive quantity is very important?
I recently decided to build a 4 disk RAIDz2 array using 4 WD30EZRX drives.
Doing some research I have found that they are 4k drives like discussed here.
Based on this, am I OK with 4 drives in my array, or should I add a 5th for performance, even though I don't really need the space?
Much appreciated,
Matt
Zarathustra[H];1039249962 said:
A lot of times people get hung up on the decisions that decrease performance over that of just having a working system. There are optimum vdev configs and optimum ashift values, but if you go with ashift 9 instead of 12 the world doesn't end and the milk in your fridge won't spoil.
For 4K drives set ashift 12. Number of disks per vdev is something like this.
RZ1 3,5,9
RZ2 4,6,10
RZ3 5,7,11