another question on RAIDZ2

[bao__zhe]

I once heard that the optimal setup for ZFS RAIDZ2 is 10 drives(8data+2parity). What is the rationale behind this?
Also i have a RAID/HBA card with 8 ports, if i setup 10 drives i will need another RAID/HBA card. Will there be any performance hit to it?

 

[HobartTas]

Greetings

Rationale is very simple as recordsize is a power of 2 and the default is 128KB but can be 512 bytes, 1,2,4,8,16,32,64..... KB and in the new version of Solaris can even be up to 2MB, so with the number of data drives also being a power of 2 as in 2,4, or your case 8 drives then dividing one into the other gives the amount of data written to each drive as (you guessed it!) a power of 2 also, this is best demonstrated by an example e.g. default 128KB / 8 data drives = 16KB written to each drive which is an exact multiple of 512/4K sectors, on the other hand lets say you had a total of 8 drives for raid-z2 (6 data and 2 parity) this would mean that 128KB / 6 = 21.33 KB so you would have wasted space as this would have to be rounded up to fill 21.5KB in 512 byte sectors or 24KB in 4K sectors. If you search this forum Sub.Mesa expounded further on this topic a year or two back. Performance degradation for not using x^2 data drives from people that have experimented with both categories of drive numbers is on the order of about 5%-10% so basically not a big deal. If you can have exactly X^2 data drives it is the most efficient scenario.

As far as your second question goes this also is a non-issue as ZFS cares only for ports and does not really care as to how they are provided. e.g. I have a 4 sata port motherboard with a 8 port 3081E-R raid card running a 10 drive Raid-Z2 using HD154UI's, OS is on 1 sata port, 2nd is currently free but I may mirror the OS later, two of the drives are on ports 3 & 4 and the remaining 8 are on the raid card, ZFS is completely unfazed by this arrangement and works without any problems, the only performance problems may be if there is a bottleneck speed wise in any one particular port holding up the whole array, e.g. when I was first experimenting with my array I first used a motherboard with only two sata ports together with the raid card, since I needed a total of 11 ports and had only 10 available I used a additional 4 port SIL3124 PCI card to supply the 11th port, even this arrangement worked without a noticible slowdown or problems (my network was only 100Mb speed at the time), realistically nobody would be using such a low end solution these days but at least it worked more or less, occasionally it would get the odd data error on the PCI card but that was about it, since I knew the arrangement was OK I just cleared the error messages and kept going as ZFS would fix any blocks that may have been written incorrectly.

I hope this answers your questions satisfactorily

Cheers

 

[bao__zhe]

thanks a lot for the info! I found this thread by sub.mesa which i think is what you are referring to:

http://hardforum.com/showthread.php?t=1546137

that's a whole lot of readings but very informative as well

so here are some additional questions:

1. by "new version of Solaris" do you mean Solaris 11.1 only?

2. has Solaris 11/11.1 improved on 4K drive handling? i.e. will it automatically detect the optimal sector size so it will apply ashift=12 and make all writing in multiple of 4K?

3. if, for example, i use 8 drives for RAIDZ2 instead of 10 drives, will the some (small) space be wasted on the drives?

4. regarding ports, i remember it is suggested to present RAIDZ components uniformly to ZFS is best for performance. i'm currently having a AOC-USAS2-L8I but is planning to buy a ServeRAID M1015. so that will not be uniform. as such will there be any performance hit?

 

[HobartTas]

by "new version of Solaris" do you mean Solaris 11.1 only?

I believe I was reading the release notes of the initial Solaris 11 11/11 where it mentioned the increase in recordsize, I thought it was 2MB but since I can't find any mention of it on the net I must have gotten confused and it appears its only 1MB, sorry about that, have a read of this article here where it mentions it.

http://www.pythian.com/news/28579/solaris-11-first-impressions-as-storage-server/

There is one aspect that I am unsure of and that is "Keep in mind that split-block storage only works for the very first block. For files of 2 or more blocks (records) it’s always rounded up. So with 1MB recordsize, a 4K file takes 4K. However a 1.1MB file takes 2 MB of space. I would imagine the use case for this new recordsize is for large file storage and would be especially useful to maximize compression (to be tested soon as well)." because if you look at page 18 of this here

http://www.google.com.au/url?sa=t&r...3oGIBw&usg=AFQjCNERgMcWgFg5cZ0gdjEQrkhvzijI1g

it implies that it reduces the height of the stripe so even if it is a 1MB recordsize (say 128KB per drive over a 10 drive raid-z2 (1MB /8 drives = 128KB) if you wrote out one byte only it should be at worst one stripe only one sector high amounting to a total of 10 sectors written, however, it should also be at best (if you did write out only that one byte of data) write out only 3 sectors which should be one data sector and 2 parity sectors (due to the Z2 parity).

I would have assumed that if there was some data that didn't fit the recordsize it would be truncated down, but the article implies it is only done where the entire file's size happens to be less than the recordsize amount. None of this would greatly affect someone using a home NAS with mostly large files but this would be catastrophic for someone setting a high recordsize for something like an office SQL database using something like 4K blocks.

A further complication is another comment that I read that I can't locate is that ZFS only writes variable height stripes (or was it variable length stripes) only if compression was turned on, so I am a bit confused myself as to how exactly ZFS packs data into sectors and records.

2. has Solaris 11/11.1 improved on 4K drive handling? i.e. will it automatically detect the optimal sector size so it will apply ashift=12 and make all writing in multiple of 4K?QUOTE]

Sorry, I have no idea as this is not a topic I am greatly familiar with as I have not used any 4K drives with ZFS at all, as far as I understand it ZFS will happily use 512 drives with obviously no problems and also there would be no problems whatsoever with ZFS and native 4K drives but from what I have seen there aren't any in existence at the moment. All the problems stem from the current 4K/512 emulation drives.

The ideal situation would be if hard drive makers had a utility to disable the 512 byte emulation altogether on the drives, so if you could take say a 2TB hard drive which is 4K internally and presents as 4 billion 512 sectors, switch it off and then have a 2TB drive with 500 million 4K sectors then this would be the best situation all round for ZFS usage as all the problems would then disappear, I can't see why they can't do this as they don't seem to have any problems fiddling with the TLER and APM settings on the drives to everyone's detriment (I refer to problems people have with hardware raid dropping consumer drives).

In any case have a read of this thread where people have actually used these drives as this may be of some help to you.

http://forums.overclockers.com.au/showthread.php?t=984341

3. if, for example, i use 8 drives for RAIDZ2 instead of 10 drives, will the some (small) space be wasted on the drives?

There would be some as detailed in my initial reply, but if you take into account what I have now posted regarding variable stripe width and height it's really hard to tell how much that would be, also with a non standard drive number I can't offer any advice as to whether setting a higher or lower size recordsize would mitigate problems in this regard, I still think compression would be of more use in this situation as well.

The best advice I could give you would be to set "compression=on" for all pools primarily because the default compression level has a fairly low overhead and should not affect perfomance much.

I suppose if you really want to know for sure you could experiment and you could set up two filesystems, one with a low recordsize and one with a high recordsize and copy say the contents of another windows machine's harddrive into them (due to varying sizes and numbers of the files stored on it), then you could see how much space is used for both and choose accordingly.

regarding ports, i remember it is suggested to present RAIDZ components uniformly to ZFS is best for performance. i'm currently having a AOC-USAS2-L8I but is planning to buy a ServeRAID M1015. so that will not be uniform. as such will there be any performance hit?

I very much doubt it, I can only perceive a performance hit if bandwidth is limited for some of the drives in the array and not others, e.g. lets say you are writing out a 4GB linux iso so ZFS is then doing full stripe writes on a 10 drive Raid-Z2 on a continuous basis where 8 drives are on the M1015 and say 2 drives are on a PCI card which has a total bandwidth of 133MB's, so that would mean the PCI would bottleneck the two drives at 66MB's each and the others would also be limited to 66MB's, so the amount of traffic in total would be 660 MB's and this would amount to 8 x 66MB's or 528 MB's of data. The only other possible problem could be if latency in replying to I/O requests is greatly different between controllers but this is not likely between two dissimilar SAS controllers but probably would be if one is a SAS PCI-e x8 and the other is a simple PCI card.

If your talking about some drives on the M1015 and some on the AOC, then no as the cards are what PCIe x8 so 8 lanes at 255MB's per lane equals 4GB's so in that case your limited by the hard drives media speed which is what 75-150 MB's each or 1.5GB's in total.

It might be a different story if you said you were doing a Raid-Z2 with 10 Sata 3 SSD's that could read/write 500MB's each then yes you might not get peak speeds if some of the drives are throttled slightly compared to the others, definitely not a problem if this is for a home NAS that might only have a gigabit link connecting it (80-100 MB's) but could be a problem for an office machine that could have a 56 Gb's infiniband link (or fiber channel or 10Gbe) connected to it.

Anyway that's the best I can answer your questions at this point, I hope its of some help

Cheers

 

[bao__zhe]

this is very helpful! i'll have more reading to do now...