ZFS storage server - design question

[Jerkson]

I have been reading (what feels like) endlessly regarding OI & ZFS for use as my home SAN/NAS.

Background:
I have been using an Adaptec 5805 with the HP SAS expander in a Habey ESC-4201C enclosure.
The Adaptec only worked at 1.5gbps with that expander and I got tired of being locked down to one specific controller. So I'm looking at a ZFS solution. I have already purchased the LSI-SAS9211-8i as a standard HBA (flashed to IT mode)

Right now my hardware setup is as follows:
-Dell Poweredge 2950 III (Xeon - 8cores @ 2.4ghz, 16gb, 2x300GB 15k SAS for VM's)
-Habey ESC-4201C ESXI 4.1 whitebox (Xeon - 4cores @ 2.0ghz, 8gb ECC, 8xhitachi 2tb 7k3000)

On the Habey/Norco ESXI box I have a virtual file server (Win2k8r2) which I pci-passthru the Adaptec card to. On this server I have one large data volume (D formatted on the Raid6 array , consisting of 8x2TB hitachi, for a total of 10.8TB of usable space. All my permissions are managed on this server using Active Directory.

The structure of the large array share is as follows:
\media
\ISO
\Machine_Backups
\homedrives

What I had hoped to do was make the Norco enclosure a physical ZFS SAN (or maybe still virtual, so I can run other stuff on it such as pfSense) using OpenIndiana and using napp-it for most management functions. In preparation I have copied ALL my data off the raid 6 array and I'm free to wipe all of those drives. I also have 3 more unused 2TB hitachi's for a total of 11.

Questions:
1) ZFS/OI - Should I create a raidz3 vdev consisting of 11x2tb (128/8 = good) knowing full well that the only way to expand this pool in the future is to add another group of 11x2tb drives?
OR should I make a 10x2tb raidz2 with a hot spare?
Or perhaps another configuration can be suggested. I REALLY want to make sure I am as protected as reasonably possible against disk failure while maintaining at least 10.8TB of usable space.

2)Fileserver - Can this large pool be presented via ISCSI to the Win2k8r2 fileserver for formatting and permissions management similar to how I am doing it now? Can that pool be sliced up and presented as multiple iscsi targets? Maybe 1 for VM's, another for the fileserver?

Any suggestions or wisdom you could pass on would be greatly appreciated. In the meantime I'm going to go scour more threads on the topic.

 

[iroc409]

Questions:
1) ZFS/OI - Should I create a raidz3 vdev consisting of 11x2tb (128/8 = good) knowing full well that the only way to expand this pool in the future is to add another group of 11x2tb drives?
OR should I make a 10x2tb raidz2 with a hot spare?
Or perhaps another configuration can be suggested. I REALLY want to make sure I am as protected as reasonably possible against disk failure while maintaining at least 10.8TB of usable space.

2)Fileserver - Can this large pool be presented via ISCSI to the Win2k8r2 fileserver for formatting and permissions management similar to how I am doing it now?

Unless I'm misinterpreting it, I think 11 is too many disks. Most of what I read says stay at or below 9 per vdev. I also read an interesting conversation recently here about the levels of raidz, and raidz3 was definitely the sweet spot, with raidz2 possibly introducing a higher risk than raidz1.

You might consider a pair of vdevs with fewer disks, but I'm sure some better experts can chime in.

I have a question though. If you export the file system as an iSCSI target, and the machine using it is Server (as you suggest), and it formats it for its use, do you still get checksum protection on the files? How is ZFS going to know the files are corrupt?

 

[Billy_nnn]

ZFS/OI - Should I create a raidz3 vdev consisting of 11x2tb (128/8 = good) knowing full well that the only way to expand this pool in the future is to add another group of 11x2tb drives?

That may be the most obvious way, but it's by no means the only way.


OR should I make a 10x2tb raidz2 with a hot spare?

Personally I'd be happy with raidz2, but it's really a personal decision!
It's always a tradeoff - performance vs resilience vs cost - bottom line is that raidz3 is more resilient, but has lower performance and higher cost - depends on your level of fear of data loss I suppose

As for the hotspare, again it's a tradeoff. Personally, for a home server, I'd keep it on the shelf as a spare and avoid any wear and tear on it, though I suppose if your server is on 24x7 and unattended, then a hotspare might be a good idea - if you tend to keep pretty much on top of things though, then personally I don't think a hotspare is really necessary on a home server.



Or perhaps another configuration can be suggested. I REALLY want to make sure I am as protected as reasonably possible against disk failure while maintaining at least 10.8TB of usable space.

With 11x 2TB drives (really 1.8TB drives), the only way to keep at least 10.8TB usable AND maintain resilience is by using one of the raidz flavours. Mirrors would be too costly on space - and as it seems you want to keep to powers of 2 for the I/O split (ie 128/8), then ideally you'd need 8x mirror pairs (so 16 drives in total).

 

[Jerkson]

Unless I'm misinterpreting it, I think 11 is too many disks. Most of what I read says stay at or below 9 per vdev. I also read an interesting conversation recently here about the levels of raidz, and raidz3 was definitely the sweet spot, with raidz2 possibly introducing a higher risk than raidz1.

You might consider a pair of vdevs with fewer disks, but I'm sure some better experts can chime in.

Thanks for your input.

I hadn't seen anything that said stay at <=9 disks per vdev. However, having multiple vdevs should increase my IOPS for the pool.

I dont see how raidz2 could possibly introduce a higher risk than raid1 since you can survive 2 simultaneous drive failures in the vdev.However, if you have a reference for that it would make for an interesting read.

If I were going with multiple vdevs and I have my 11 disks to work with what would you suggest? no combination of that will divide 128 nicely. I could get a 12th drive and do
2 raidz2 vdevs of 6 drives

I have a question though. If you export the file system as an iSCSI target, and the machine using it is Server (as you suggest), and it formats it for its use, do you still get checksum protection on the files? How is ZFS going to know the files are corrupt?

That was something I kept asking myself as well as it would see the NTFS filesystem as one large 'object' so to speak. So instead I should share it as CIFS or SMB...but then how to I manage my permissions still using AD?

 

[Jerkson]

ZFS/OI - Should I create a raidz3 vdev consisting of 11x2tb (128/8 = good) knowing full well that the only way to expand this pool in the future is to add another group of 11x2tb drives?

That may be the most obvious way, but it's by no means the only way.

Yes, I definitely mis-spoke here (or is it mis-typed?) It is not the only way..I could also swap out one drive at a time and resilver... or create a whole new pool with new drives and migrate the data. I'm sure there are others.

OR should I make a 10x2tb raidz2 with a hot spare?

Personally I'd be happy with raidz2, but it's really a personal decision!
It's always a tradeoff - performance vs resilience vs cost - bottom line is that raidz3 is more resilient, but has lower performance and higher cost - depends on your level of fear of data loss I suppose

Well....Ive been fine with raid6 thus far so I'm comfortable with raidz2 going forward, especially since losing data would mean two simultaneous drives within the same vdev would have to die for the pool to go dead. Going forward when I have 2 vdevs in the pool it could withstand 4 drive failures total before catastrophe. Im comfortable with that. In the end I would liek to get enough read throughput to saturate a 2gbps LACP connection but more would always be nice

As for the hotspare, again it's a tradeoff. Personally, for a home server, I'd keep it on the shelf as a spare and avoid any wear and tear on it, though I suppose if your server is on 24x7 and unattended, then a hotspare might be a good idea - if you tend to keep pretty much on top of things though, then personally I don't think a hotspare is really necessary on a home server.
{/quote]

This is a good point but I can just see it with my luck that I'm on a week's vacation or at a conference and drives start dying on me. Yes, the server is on 24/7 but I stay on top of things.

Or perhaps another configuration can be suggested. I REALLY want to make sure I am as protected as reasonably possible against disk failure while maintaining at least 10.8TB of usable space.

With 11x 2TB drives (really 1.8TB drives), the only way to keep at least 10.8TB usable AND maintain resilience is by using one of the raidz flavours. Mirrors would be too costly on space - and as it seems you want to keep to powers of 2 for the I/O split (ie 128/8), then ideally you'd need 8x mirror pairs (so 16 drives in total).

In the example of 8x mirror pairs, If 2 drives died and happened to be both drives from a pair the whole pool is gone, correct? Im sure this would have the best IO though

 

[iroc409]

Thanks for your input.

I hadn't seen anything that said stay at <=9 disks per vdev. However, having multiple vdevs should increase my IOPS for the pool.

I dont see how raidz2 could possibly introduce a higher risk than raid1 since you can survive 2 simultaneous drive failures in the vdev.However, if you have a reference for that it would make for an interesting read.

If I were going with multiple vdevs and I have my 11 disks to work with what would you suggest? no combination of that will divide 128 nicely. I could get a 12th drive and do
2 raidz2 vdevs of 6 drives

The failure rate was a probability of drive error calculation. It was in a thread here, and it was very mathematically involved LOL. Anyway, the person who was doing it came up with a failure probability higher in Raidz2 than z1 & z3. Has to do with individual drive failure probabilities compounded into multiple drives (why two disks are more failure-prone than a single disk).

I've read on here not to go over 9 disks, I think basically performance degrades when you get too large a vdev. But, I looked through the ZFS Best Practices guide here (http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide), which hints at not using more than 9 disks:

RAIDZ Configuration Requirements and Recommendations

A RAIDZ configuration with N disks of size X with P parity disks can hold approximately (N-P)*X bytes and can withstand P device(s) failing before data integrity is compromised.

Start a single-parity RAIDZ (raidz) configuration at 3 disks (2+1)
Start a double-parity RAIDZ (raidz2) configuration at 6 disks (4+2)
Start a triple-parity RAIDZ (raidz3) configuration at 9 disks (6+3)
(N+P) with P = 1 (raidz), 2 (raidz2), or 3 (raidz3) and N equals 2, 4, or 6
The recommended number of disks per group is between 3 and 9. If you have more disks, use multiple groups.

That was something I kept asking myself as well as it would see the NTFS filesystem as one large 'object' so to speak. So instead I should share it as CIFS or SMB...but then how to I manage my permissions still using AD?

Hopefully someone could answer this, as I'd really like to run this setup.

If you haven't read it, the ZFS Evil Tuning Guide is a good read, too.
http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide

 

[Jerkson]

So max of 9 it is.

Looks like in my example, getting a 12th drive and doing two 6 drive raidz2 vdevs is my best bet.
8 useable drives.
My only other option is 3 3disk raidz vdevs but I dont like the idea of that.

 

[brutalizer]

I am doing something similar. But I am planning to use Raidz3 with 11 disks. I mean, 9 disks or 11, what is the difference. 11 disks are 8 + 3, which is supposedly good for ZFS.

With 8 disks you get 800MB/sec or so. Let us assume performance decreases with 11 disks, so what? Maybe I get 600MB/sec? I can live with that. I am concerned with safety. Anything above 300MB/sec suffices. Heck, even 100MB/sec would suffice - if my data was safe.

And raidz3 is safest. Thus, 11 disks it is. For me.

I am going to use
http://store.sansdigital-shop.com/totr8baysass6.html
$400 via Amazon.

The rest of the 3 disks, I put in my computer chassi. I have an on/off switch connected to the disk power cable. The disk cable powers all three disks. Thus, I have a small PC-V700 full-ATX case. I just hide the disk chassi when I am not needing the array. And turn it on, when I need it. I will have a 3TB disk as a work space. When it is full, I turn on the array and save everything to the array. Also, an SSD as a boot disk.

 

[Jerkson]

I am doing something similar. But I am planning to use Raidz3 with 11 disks. I mean, 9 disks or 11, what is the difference. 11 disks are 8 + 3, which is supposedly good for ZFS.

With 8 disks you get 800MB/sec or so. Let us assume performance decreases with 11 disks, so what? Maybe I get 600MB/sec? I can live with that. I am concerned with safety. Anything above 300MB/sec suffices. Heck, even 100MB/sec would suffice - if my data was safe.

And raidz3 is safest. Thus, 11 disks it is. For me.

Where did you get the figure of 800MB/sec? I assume youre taking the avg throughput of a drive (100MB/sec) and multiplying it by the number of data drives.
ZFS will essentially stripe across all disks in a vdev and stripe across all vdevs in a pool. Your setup will work much like a 11 disk raid 5 or 6 would (striping across all data disks) yet you would never realize 800MB/sec in that scenario...not even close.

 

[Jerkson]

oh and iroc...I found the mathematical proof of failure rates you were referring to. It was calculated for any 8-data drive array. Meaning an 11 drive raidz3, 10 drive raidz2, 9 drive raidz or 16 drive mirror.

RAID-10:
F = 5%

RAID-Z1:
1 - (1 - F)^(9 - 1) = 33.7%

RAID-Z2:
1 - (1 - F)^(10 - 1) - (10 - 1) F (1 - F)^(10 - 2) = 7.1%

RAID-Z3:
1 - (1 - F)^(11 - 1) - (11 - 1) F (1 - F)^(11 - 2) - (11 - 1)(11 - 2) F^2 (1 - F)^(11 - 3) / 2
= 1.15%

 

[_Gea]

Questions:
1) ZFS/OI - Should I create a raidz3 vdev consisting of 11x2tb (128/8 = good) knowing full well that the only way to expand this pool in the future is to add another group of 11x2tb drives?
OR should I make a 10x2tb raidz2 with a hot spare?
Or perhaps another configuration can be suggested. I REALLY want to make sure I am as protected as reasonably possible against disk failure while maintaining at least 10.8TB of usable space.

2)Fileserver - Can this large pool be presented via ISCSI to the Win2k8r2 fileserver for formatting and permissions management similar to how I am doing it now? Can that pool be sliced up and presented as multiple iscsi targets? Maybe 1 for VM's, another for the fileserver?

Any suggestions or wisdom you could pass on would be greatly appreciated. In the meantime I'm going to go scour more threads on the topic.

1.
ZFS is very flexible. You can do a lot of useless or uneconomic things.
Beside that, its mainly a question of optimisation to a specific workload or aim

example
With 11 disks you may create a pool from
one vdev with 11 basic disks (fast but useless)
one vdev with 11 disks in raid z3 (very good for a backup server, slow)
one vdev with 10 disks in raid-z2 + hotspare (useless with a raid z3 you have the same with a hot hotspare)
two vdevs with 5 disks each + hotspare (less capacity, faster)
5 x mirror + hotspare (typical use for I/O heavy load like a ESXi datastore, lower capacity, good security)
3 x 3 mirror + 2 hotspare (ultra high speed + ultra high data security, lowest capacity)

I would also not care too much about these 'golden' numbers of disks in a vdev to build perfect balanced pools.
ZFS is good enough to do its best and mostly the difference is irrelevant in real world.
Only thing to care: you cannot remove a vdev, so some constructs may be ineffective. The seond is, you should never
lower the overall redundancy of a pool. (like: never add a basic vdev to a RaidZ pool or add a raid-Z to a raid-Z2)

2.
You can present parts or the whole pool via iSCSI to a Win2008 server. But why?
Solaris + an external AD Server is a perfect SMB filer. I was happy to replace all my Windows 2008 filer with Solaris,
no more weekly security fixes/reboots, no more problems with path length, no more problem with users removing admin from
permissions, no more licence cost,....., AND access to hundreds of ZFS snaps via previous version - a server without hassle

 

[Jerkson]

1.
ZFS is very flexible. You can do a lot of useless or uneconomic things.
Beside that, its mainly a question of optimisation to a specific workload or aim

example
With 11 disks you may create a pool from
one vdev with 11 basic disks (fast but useless)
one vdev with 11 disks in raid z3 (very good for a backup server, slow)
one vdev with 10 disks in raid-z2 + hotspare (useless with a raid z3 you have the same with a hot hotspare)
two vdevs with 5 disks each + hotspare (less capacity, faster)
5 x mirror + hotspare (typical use for I/O heavy load like a ESXi datastore, lower capacity, good security)
3 x 3 mirror + 2 hotspare (ultra high speed + ultra high data security, lowest capacity)

I would also not care too much about these 'golden' numbers of disks in a vdev to build perfect balanced pools.
ZFS is good enough to do its best and mostly the difference is irrelevant in real world.
Only thing to care: you cannot remove a vdev, so some constructs may be ineffective. The seond is, you should never
lower the overall redundancy of a pool. (like: never add a basic vdev to a RaidZ pool or add a raid-Z to a raid-Z2)

2.
You can present parts or the whole pool via iSCSI to a Win2008 server. But why?
Solaris + an external AD Server is a perfect SMB filer. I was happy to replace all my Windows 2008 filer with Solaris,
no more weekly security fixes/reboots, no more problems with path length, no more problem with users removing admin from
permissions, no more licence cost,....., AND access to hundreds of ZFS snaps via previous version - a server without hassle

Thanks so much for chiming in (I hoped you would )

I would have up to 3 media streamers simultaneously playing back HD video as well as 1 PC using it as a filestore for smaller documents & critical data, and a Acronis True Image backup store with weekly disk dumps from 3 client PC's. So speed isnt "really" a concern but its nice when I need to dump a PC image that it can receive it fast enough to saturate a gigabit connection (~120MB/sec write)

Does that seem possible using a 11 disk raidz3? I would hope so. What would you do if you had 11 drives in my scenario?

 

[_Gea]

raid z3

my 3 Backupsystems have 15 x 2 TB pools from one Raid Z3 vdev and 8-16 GB RAM
Performance is good. Only poblem: more than a day resilver time. but with raid z3 this is ok.

(i use 15 disks because this is the max with my 16 bay enclosures. I wanted to keep one bay free
for disk replacements)

 

[dougal1957]

raid z3

my 3 Backupsystems have 15 x 2 TB pools from one Raid Z3 vdev and 8-16 GB RAM
Performance is good. Only poblem: more than a day resilver time. but with raid z3 this is ok.

(i use 15 disks because this is the max with my 16 bay enclosures. I wanted to keep one bay free
for disk replacements)

Can I jump in here and ask GEA for advice on best way to utilise 12 x 1.5 TB drives and 8 X 2 TB drives (the 2 TB will need to be added to the pool a little later once I have managed to copy data from them to the 1.5's)

The data is purely for Media streaming to HD Media player arround the house.

These drives will be in a Norce 4220 Chassis but will be adding a Further 16 drive chasis to the system at a later date when funds to purchase the drives allow.

I had thought of using 2 vdevs of RaidZ2 using all of the drives and then pooling them. Is this a sensible way of doing this or are there better options to me (storage capacity and some resilience are the prime concerns here as most of the data can be recovered but at a huge cost in time).

TIA Doug

 

[_Gea]

The idea of a datapool is, throw in whatever you have and your datasets can grow up to the new size.
Only problem. You cannot remove after you have added so overall usage may not be very economic.

with 12 disks i would build a raid-z3 vdev. and add a raid-z2 from 8 disks later.
think of a common hotspare

 

[dougal1957]

The idea of a datapool is, throw in whatever you have and your datasets can grow up to the new size.
Only problem. You cannot remove after you have added so overall usage may not be very economic.

with 12 disks i would build a raid-z3 vdev. and add a raid-z2 from 8 disks later.
think of a common hotspare

So would I be correct in saying that a common hot spare would need to be at least as big as the largest drive in the Pool?

Doug

 

[brutalizer]

raid z3

my 3 Backupsystems have 15 x 2 TB pools from one Raid Z3 vdev and 8-16 GB RAM
Performance is good. Only poblem: more than a day resilver time. but with raid z3 this is ok.

(i use 15 disks because this is the max with my 16 bay enclosures. I wanted to keep one bay free
for disk replacements)

What perf do you get with 15 disk raidz3?

 

[_Gea]

What perf do you get with 15 disk raidz3?

enough for a backupserver with mainly sequential writes
example:

write 20.48 GB via dd, please wait...
time dd if=/dev/zero of=/backup3/dd.tst bs=1024000 count=20000

20000+0 records in
20000+0 records out

real     1:06.8
user        0.0
sys        17.6

20.48 GB in 66.8s = 306.59 MB/s Write

read 20.48 GB via dd, please wait...
time dd if=/backup3/dd.tst of=/dev/null bs=1024000

20000+0 records in
20000+0 records out

real       41.4
user        0.0
sys        31.7

20.48 GB in 41.4s = 494.69 MB/s Read

System:
Memory size: 4095 Megabytes

System Configuration: Supermicro X7DB8
BIOS Configuration: Phoenix Technologies LTD 2.1a 12/20/2008
BMC Configuration: IPMI 1.0 (unknown)

==== Processor Sockets ====================================

Version                          Location Tag
-------------------------------- --------------------------
Intel(R) Xeon(TM) processor      LGA771/CPU1

NAME        STATE     READ WRITE CKSUM     CAP            Product
	backup3     ONLINE       0     0     0
	  raidz3-0  ONLINE       0     0     0
	    c3t0d0  ONLINE       0     0     0     1000.20 GB     ST31000340NS
	    c3t1d0  ONLINE       0     0     0     1000.20 GB     WDC WD1002FBYS-0
	    c3t2d0  ONLINE       0     0     0     1000.20 GB     ST31000340NS
	    c3t3d0  ONLINE       0     0     0     1000.20 GB     ST31000340NS
	    c3t4d0  ONLINE       0     0     0     1000.20 GB     ST31000340NS
	    c3t5d0  ONLINE       0     0     0     1000.20 GB     ST31000340NS
	    c3t7d0  ONLINE       0     0     0     1000.20 GB     WDC WD1002FBYS-0
	    c4t0d0  ONLINE       0     0     0     1000.20 GB     WDC WD1002FBYS-0
	    c4t1d0  ONLINE       0     0     0     1000.20 GB     WDC WD1002FBYS-0
	    c4t2d0  ONLINE       0     0     0     1000.20 GB     WDC WD1002FBYS-0
	    c3t6d0  ONLINE       0     0     0     1000.20 GB     WDC WD10EACS-00D
	    c4t4d0  ONLINE       0     0     0     1000.20 GB     WDC WD1002FBYS-0
	    c4t5d0  ONLINE       0     0     0     1000.20 GB     WDC WD1002FBYS-0
	    c4t6d0  ONLINE       0     0     0     1000.20 GB     WDC WD1002FBYS-0
	    c4t7d0  ONLINE       0     0     0     1000.20 GB     WDC WD1002FBYS-0

others are quite similar

 

[Billy_nnn]

In the example of 8x mirror pairs, If 2 drives died and happened to be both drives from a pair the whole pool is gone, correct? Im sure this would have the best IO though

Yes, if 2 drives failed simultaneously and completely and they both happened to be from the same mirror pair, then you'd be looking at total loss of pool!

As for IO, it'd certainly have the highest IOPS capability - it depends what type of IO you actually do though as to whether you'd see the full benefit!


However, on rereading my post now, I think I've ended up confusing myself (not that hard ) - my understanding of the way zfs works is that records are not split across vdevs in a pool, but they are split across drives which make up each vdev (if the vdev is striped) - so the part about needing 8x mirror pairs for a "powers of 2" IO split, is actually incorrect!
Not sure what I was thinking now......

 

[Jerkson]

raid z3

my 3 Backupsystems have 15 x 2 TB pools from one Raid Z3 vdev and 8-16 GB RAM
Performance is good. Only poblem: more than a day resilver time. but with raid z3 this is ok.

(i use 15 disks because this is the max with my 16 bay enclosures. I wanted to keep one bay free
for disk replacements)

Thanks so much _Gea!

The performance number you listed are more than enough for my use-case.
And having more than a day resilver time is perfectly acceptable to me considering my 8x2TB raid6 on the Adaptec 5805 was taking up to 3 days to rebuild my raid 6, leaving me with only 1 safe failure left. In this scenario with raidz3 I will have 2 left.

I will start with a pool of 1 raidz3 vdev of 11 disks which leaves 8 drives for storage.
I'll let you know how it goes!

I've been reading into AD integration and there seems to be lots of initial configuration.
I started reading here (and the following pages in the doc)

 

[_Gea]

I've been reading into AD integration and there seems to be lots of initial configuration.
I started reading here (and the following pages in the doc)

If you only need a SMB filer with AD user, you must read this
http://docs.oracle.com/cd/E19963-01...goperationmodetm.html#configuredomainmodetask

or if you use napp-it
goto menu service - smb - active directory and join the domain

 

[Jerkson]

If you only need a SMB filer with AD user, you must read this
http://docs.oracle.com/cd/E19963-01...goperationmodetm.html#configuredomainmodetask

or if you use napp-it
goto menu service - smb - active directory and join the domain

Of course im using napp-it
I just did as you said and connected my test OI vm to my AD perfectly. I was able to add files to the SMB share from windows as a guest.

In the help under zfs folder/datasets view it says:

ACL
Set shared folder ACL and local users with napp-its's ACL extension.
If you want to modify Share-Level ACL or ACL on files and folders, you can use the nonfree ACL extension

Or do it from Windows (not all Windows versions supports remote ACL settings):
If you want to set special file or folder-ACL's for SMB, set unix permissions to 777 and connect the share as root from Windows.
(Do not forget to reset root-pw once after first installation of napp-it to create a SMB password). Right Click on a folder and set
permissions with properties-security.

I have a share with quest access on and I want to change the permissions using windows to lock it down. This cannot be done under the guest user I have the drive mapped as; permission denied.

When I try to map the ZFS SMB share as root on windows, on the credentials screen it shows the domain of "home" (as an example) but the root user is not a domain user but a local user in OI...

I have tried specifying root@OI_dns_name and root@OI_ip_address to no avail.
I even tried creating an AD account called root with the same password as the root user in OI.

Please help. Im so close

EDIT: one last question...

If my folder structure is currently like this:
homdrives
-user1
-user2
-user3
media
backups

Would it make sense that all 3 root directories are just directories on one ZFS filesystem, or should homedrives, media and backups possibly be 3 separate filesystems?

The reason I ask this is I may want to snapshot homedrives but not Media...and I believe the only way to do so would be to have 3 separate ZFS filesystems, yes?

 

[_Gea]

1. if you enable guest access, you do not need to login and can't modify/ use permissions
-> unshare the dataset and reshare with guest disabled

2. now you can connect either as a lokal user (example root) or as a domain-user

3. if you want to connect as domain user, you must allow
(default is root only, allow everyone@=modify or full)

4. snaps are per dataset
in your case, you must create multiple datasets

 

[Jerkson]

So here's where I'm at with all this. I started fresh to do a good test before deploying.

Setup:

• Spun up the new VM in Esxi4.1: 6gb RAM, 40gb disk, 2vcpu @ 2.0ghz,
• Fresh install of OI updated to 151a3.
• Install VMware tools
• PCI-Passthrou'd the sas2008-based 9211-8i in IT mode with 11x2TB Hitachi's.
• Installed Napp-it... following the manual
• Attached napp-it to AD successfully
• Created a zpool called datapool1 of one vdev (11x2tb raidz3)
• Created 3 ZFS filesystems: media, homedrives, backups

SMB is enabled for each of the ZFS datasets.

I can map a windows client to any of the datasets just fine. However, when copying a 3GB test file from the SSD of the Windows client I get a transfer rate of 72-79MB/sec. Before with my Raid6 I would saturate my gigabit line ~110MB/sec.
Copying the file back from the ZFA store to the Windows machine results in marginally better transfer rates ~85MB/sec

Moving this test file from one ZFS dataset to another only goes about 55-59MB/sec (when done from the windows client) and about 76MB/sec when done using "time cp /datapool1/media/file1.tst /datapool1/Backup/testfilebackup.tst"

Below is my output from the benchmark. Based on the benchmark it should be flying!

write 20.48 GB via dd, please wait...
time dd if=/dev/zero of=/datapool1/dd.tst bs=1024000 count=20000

20000+0 records in
20000+0 records out

real     1:36.8
user        0.2
sys        17.0

20.48 GB in 96.8s = 211.57 MB/s Write

read 20.48 GB via dd, please wait...
time dd if=/datapool1/dd.tst of=/dev/null bs=1024000

20000+0 records in
20000+0 records out

real       39.1
user        0.0
sys        20.5

20.48 GB in 39.1s = 523.79 MB/s Read

and here is my pool:

pool: datapool1
 state: ONLINE
  scan: none requested
config:

	NAME                       STATE     READ WRITE CKSUM     CAP            Product
	datapool1                  ONLINE       0     0     0
	  raidz3-0                 ONLINE       0     0     0
	    c2t5000C500452659E4d0  ONLINE       0     0     0     2.00 TB        ST2000DM001-9YN1
	    c2t5000C500452CBD30d0  ONLINE       0     0     0     2.00 TB        ST2000DM001-9YN1
	    c2t5000C500453369ABd0  ONLINE       0     0     0     2.00 TB        ST2000DM001-9YN1
	    c2t5000CCA369C350BCd0  ONLINE       0     0     0     2.00 TB        Hitachi HDS72302
	    c2t5000CCA369C44987d0  ONLINE       0     0     0     2.00 TB        Hitachi HDS72302
	    c2t5000CCA369C46BF7d0  ONLINE       0     0     0     2.00 TB        Hitachi HDS72302
	    c2t5000CCA369C46C13d0  ONLINE       0     0     0     2.00 TB        Hitachi HDS72302
	    c2t5000CCA369C46D75d0  ONLINE       0     0     0     2.00 TB        Hitachi HDS72302
	    c2t5000CCA369C47326d0  ONLINE       0     0     0     2.00 TB        Hitachi HDS72302
	    c2t5000CCA369C47981d0  ONLINE       0     0     0     2.00 TB        Hitachi HDS72302
	    c2t5000CCA369C47F1Ad0  ONLINE       0     0     0     2.00 TB        Hitachi HDS72302

errors: No known data errors

  pool: rpool
 state: ONLINE
  scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM     CAP            Product
	rpool       ONLINE       0     0     0
	  c4t0d0s0  ONLINE       0     0     0     42.95 GB       Virtual disk

errors: No known data errors

 

[Jerkson]

Any ideas? I dont want to just start tuning at random..

 

[Jerkson]

I've been running tests throughout the day and copying via NFS from windows is painfully slow @ 12MB/sec. At least SMB was quicker.

Still trying to sort out why it's so slow.

 

[_Gea]

About 80 GB/s is not so bad for a 16 TB Raid-Z3 array and only 6 GB RAM.
The dd values are for pure sequential performancr which is always much higher
than real world values where I/O becomes more important (You have I/O performance
like just one disk)

Your tuning options are mostly Raid-Level (Raid-Z3 is the slowest, multiple Raid-1 the fastest),
more RAM or a fast SSD as Read Cache.

ps
If you try NFS, you should disable sync write for the folder to get comparable values
otherwise you may check against the very secure but very slow sync write values!

 

[Jerkson]

About 80 GB/s is not so bad for a 16 TB Raid-Z3 array and only 6 GB RAM.
The dd values are for pure sequential performancr which is always much higher
than real world values where I/O becomes more important (You have I/O performance
like just one disk)

Your tuning options are mostly Raid-Level (Raid-Z3 is the slowest, multiple Raid-1 the fastest),
more RAM or a fast SSD as Read Cache.

ps
If you try NFS, you should disable sync write for the folder to get comparable values
otherwise you may check against the very secure but very slow sync write values!

Good to know about the NFS. The Sync property of the folder is set to "standard" as are the other folders. Is this what should be turned off? I read about NOT doing that...

Using an SSD as Read Cache wont speed up writes, will the RAM speed up both directions?

If this is the fastest it will get I'm a little disappointed now since my Raid card got better speeds although with one less parity drive.

Thanks again for all your help and suggestions.

 

[_Gea]

Good to know about the NFS. The Sync property of the folder is set to "standard" as are the other folders. Is this what should be turned off? I read about NOT doing that...

Using an SSD as Read Cache wont speed up writes, will the RAM speed up both directions?

If this is the fastest it will get I'm a little disappointed now since my Raid card got better speeds although with one less parity drive.

Thanks again for all your help and suggestions.

There is nothing free on earth.
Either more secure Raid levels and more secure filesystems like ZFS with End-to-End
checksums and copy on write needs more performance to get same performance like
systems without such features. The pro is that ZFS scales very well with vdevs, so you
can reach best performance values with best security.

About RAM and writes
All normal (async) large or small writes are going to RAM in ZFS where they are collected for a few seconds (ex 5s)
and then written to disk sequentially at once (multiple small write to one large sequential write conversion). Therfor RAM
can speed up reads and writes. This is how SMB works. On a powerloss up to these 5s of last data is lost.

Some programms (like ESXi over NFS) wants to reach a higher security level activating sync writes.
With sync write demands, every single write action must go to disk and commited immediatly until the next
one can occur. With small writes, you are hammering your disk. Sequential performance does not help here.
You need only good I/O values to position your disk heads and write a few bytes each time. This is a bad situation
for every file system.

If you disable sync property, ZFS handles all sync write requests like normal writes. You have full performance with
the danger of 5s dataloss on powerloss. If you want to combine high speed regular writes with the sync to async conversion,
you can use a fast ZIL Log device. If its fast enough (like a DRAM ZIL), these secure sync writes can be as fast like normal
writes with ZFS.

 

[Jerkson]

There is nothing free on earth.
Either more secure Raid levels and more secure filesystems like ZFS with End-to-End
checksums and copy on write needs more performance to get same performance like
systems without such features. The pro is that ZFS scales very well with vdevs, so you
can reach best performance values with best security.

About RAM and writes
All normal (async) large or small writes are going to RAM in ZFS where they are collected for a few seconds (ex 5s)
and then written to disk sequentially at once (multiple small write to one large sequential write conversion). Therfor RAM
can speed up reads and writes. This is how SMB works. On a powerloss up to these 5s of last data is lost.

Some programms (like ESXi over NFS) wants to reach a higher security level activating sync writes.
With sync write demands, every single write action must go to disk and commited immediatly until the next
one can occur. With small writes, you are hammering your disk. Sequential performance does not help here.
You need only good I/O values to position your disk heads and write a few bytes each time. This is a bad situation
for every file system.

If you disable sync property, ZFS handles all sync write requests like normal writes. You have full performance with
the danger of 5s dataloss on powerloss. If you want to combine high speed regular writes with the sync to async conversion,
you can use a fast ZIL Log device. If its fast enough (like a DRAM ZIL), these secure sync writes can be as fast like normal
writes with ZFS.

Perhaps I will do a comparison with a zpool of 2 raidz2 (5x2tb) just to see if throughput is better everything else remaining constant. If not, I will re-do the 11 drive raidz3 and start adding ram.

I will also passthru a physical Intel nic to the VM to see if that helps any.

 

[Jerkson]

Here's an update from my benchmarks in case anyone is interested. I know they are not scientific or anything but it gives me an idea of the performance difference between different pool configs...all other things being equal.

Copying a ~10GB MKV video from a Win 7 PC to SAn through a SMB share

Original pool: 1vdev - 11 drive raidz3 average transfer speed was 72MB/sec
2nd Pool: 2vdevs - 5 disk raidz2 each = same speed as above 72MB/sec
3rd Pool 5vdevs - 2 disk mirror pairs = Pinned to 106MB/sec constant

The mirrored pairs were hitting the upper bounds of my gigabit line if you account for protocol overheads and such. I simply did this final pool config to ensure that there wasn't some unseen variable (nic, VM nic driver, bad config, etc) that was causing the slower speeds with the other 2 pools.

So it looks like I should start by getting more RAM in this system as well as look into a ZIL cache such as an SSD (for starters)

Thanks to eberyone for their input!

 

[brutalizer]

Here's an update from my benchmarks in case anyone is interested. I know they are not scientific or anything but it gives me an idea of the performance difference between different pool configs...all other things being equal.

Copying a ~10GB MKV video from a Win 7 PC to SAn through a SMB share

Original pool: 1vdev - 11 drive raidz3 average transfer speed was 72MB/sec
2nd Pool: 2vdevs - 5 disk raidz2 each = same speed as above 72MB/sec
3rd Pool 5vdevs - 2 disk mirror pairs = Pinned to 106MB/sec constant

The mirrored pairs were hitting the upper bounds of my gigabit line if you account for protocol overheads and such. I simply did this final pool config to ensure that there wasn't some unseen variable (nic, VM nic driver, bad config, etc) that was causing the slower speeds with the other 2 pools.

So it looks like I should start by getting more RAM in this system as well as look into a ZIL cache such as an SSD (for starters)

Thanks to eberyone for their input!

Every config you tested gives more than 110MB/sec (which is the max for a gigabit line). Thus, all of them should hit close to 110MB/sec. Something might be weir in your config.

It would be interesting if you also did a benchmark on each zpool, with "dd if=/..." and "dd of=/..."

 

[Jerkson]

Every config you tested gives more than 110MB/sec (which is the max for a gigabit line). Thus, all of them should hit close to 110MB/sec. Something might be weir in your config.

It would be interesting if you also did a benchmark on each zpool, with "dd if=/..." and "dd of=/..."

Every config I tested yields 110MB/sec? I'm assuming you mean in dd tests...

as _Gea said above regarding dd:

The dd values are for pure sequential performancr which is always much higher
than real world values where I/O becomes more important (You have I/O performance
like just one disk)

Indeed local dd tests are all well above 110MB/sec.

"zdb datapool1 | grep ashift" returns ashift=12. Shouldn't this be 9 with non 4k drives? How is this number automatically chosen in napp-it?

I used the default napp-it setup. Here is the output of zfs get all datapool1/Media

NAME	 PROPERTY	 VALUE	 SOURCE
 datapool1/Media	 type	 filesystem	 -
 datapool1/Media	 creation	 Mon Apr 23 21:22 2012	 -
 datapool1/Media	 used	 1.56T	 -
 datapool1/Media	 available	 10.5T	 -
 datapool1/Media	 referenced	 1.56T	 -
 datapool1/Media	 compressratio	 1.00x	 -
 datapool1/Media	 mounted	 yes	 -
 datapool1/Media	 quota	 none	 default
 datapool1/Media	 reservation	 none	 default
 datapool1/Media	 recordsize	 128K	 default
 datapool1/Media	 mountpoint	 /datapool1/Media	 default
 datapool1/Media	 sharenfs	 off	 default
 datapool1/Media	 checksum	 on	 default
 datapool1/Media	 compression	 off	 local
 datapool1/Media	 atime	 off	 local
 datapool1/Media	 devices	 on	 default
 datapool1/Media	 exec	 on	 default
 datapool1/Media	 setuid	 on	 default
 datapool1/Media	 readonly	 off	 default
 datapool1/Media	 zoned	 off	 default
 datapool1/Media	 snapdir	 hidden	 local
 datapool1/Media	 aclmode	 passthrough	 local
 datapool1/Media	 aclinherit	 passthrough	 local
 datapool1/Media	 canmount	 on	 default
 datapool1/Media	 xattr	 on	 default
 datapool1/Media	 copies	 1	 default
 datapool1/Media	 version	 5	 -
 datapool1/Media	 utf8only	 on	 -
 datapool1/Media	 normalization	 formD	 -
 datapool1/Media	 casesensitivity	 mixed	 -
 datapool1/Media	 vscan	 off	 default
 datapool1/Media	 nbmand	 off	 local
 datapool1/Media	 sharesmb	 name=Media	 local
 datapool1/Media	 refquota	 none	 default
 datapool1/Media	 refreservation	 none	 default
 datapool1/Media	 primarycache	 all	 default
 datapool1/Media	 secondarycache	 all	 default
 datapool1/Media	 usedbysnapshots	 0	 -
 datapool1/Media	 usedbydataset	 1.56T	 -
 datapool1/Media	 usedbychildren	 0	 -
 datapool1/Media	 usedbyrefreservation	 0	 -
 datapool1/Media	 logbias	 latency	 default
 datapool1/Media	 dedup	 off	 default
 datapool1/Media	 mlslabel	 none	 default
 datapool1/Media	 sync	 standard	 default
 datapool1/Media	 refcompressratio	 1.00x	 -
 datapool1/Media	 written	 1.56T	 -

EDIT: After looking into the ashift=12 I realized something...

3 of the drives in my array are New Seagate 1TB/platter 2TB drives (ST2000DM001-9YN1) which are going to be systematically replaced with more 2tb hitachi's when they come in shortly. I just realized that they are 4k drives that do 512 emulation. Perhaps it is detected that they are 4k and the pool is created with ashift=12 instead of 9. In my tests, I created the vdevs starting with the bottom of the disks lists (hitachi's), maybe the ashift was set differently in the mirrored option before. However, if dd tests are all well above 200MB /sec write so regardless of any of this I should still be able to saturate a gig line.

 

[_Gea]

ZFS creates vdevs with the ashift value of 12 automatically if the disks are reporting 4k.

Only problem: 4k disks lying about beeing 512b ones with the effect of a "wrong" ashift 9
resulting in a lower performance.

 

[Jerkson]

ZFS creates vdevs with the ashift value of 12 automatically if the disks are reporting 4k.

Only problem: 4k disks lying about beeing 512b ones with the effect of a "wrong" ashift 9
resulting in a lower performance.

Well it looks like these drives didn't lie well enough
The pool is set to ashift=12.

From the reading I have done using ashift =12 (even on non-4k drives_ would be preferred especially if there is a chance you would use 4k drives in the future.
There should be no performance penalty from, using ashift=12 on 512b drives, and this way I am future-proofed, so to speak.

 

[Jerkson]

I figured it was better to revive this thread than to create a new one....so here goes.

Performance over the last week has been pretty good at 80MB/sec transfers and 340MB/sec resilvers (replaced the 3 Seagates with Hitachi's)
I checked and all my drives are at firmware A180 as they have been since they were purchased back in Dec 2010 and I'm not sure there would be any benefit to going to v800 f/w so Im going by the mentality of "if it ain't broke, don't fix it"

However, during a scrub I noticed that the disk activity is very high on only a few drives....see "screenshot"

root@JJ-FS01:~# iostat -nx
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 fd0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c3t0d0
    0.2    0.2    1.0    2.1  0.0  0.0    0.0    1.9   0   0 c4t0d0
    0.0    0.1    0.8    0.7  0.0  0.0    0.0    2.9   0   0 c2t5000C500452CBD30d0
  296.1    2.0 7023.3   10.0  0.0  0.2    0.0    0.7   0   9 c2t5000CCA369C47981d0
  296.4    2.0 7020.3   10.0  0.0  0.2    0.0    0.7   0   9 c2t5000CCA369C46C13d0
  102.0    0.5 2335.5    2.8  0.0  0.1    0.0    0.6   0   3 c2t5000C500452659E4d0
  296.8    2.0 7020.8   10.0  0.0  0.2    0.0    0.7   0   9 c2t5000CCA369C46D75d0
  296.5    2.0 7021.9   10.0  0.0  0.2    0.0    0.7   0   9 c2t5000CCA369C47326d0
  295.8    2.0 7020.7   10.0  0.0  0.2    0.0    0.7   0   9 c2t5000CCA369C46BF7d0
  295.4    2.0 7021.0   10.0  0.0  0.2    0.0    0.7   0   9 c2t5000CCA369C44987d0
  296.0    2.0 7023.8   10.0  0.0  0.2    0.0    0.7   0   9 c2t5000CCA369C47F1Ad0
  204.4    1.0 4658.3    5.0  0.0  0.1    0.0    0.6   0   5 c2t5000C500453369ABd0
  294.7    2.0 7021.5   10.0  0.0  0.2    0.0    0.7   0   9 c2t5000CCA369C350BCd0
  532.9  158.7 12642.6 6203.9  0.0  1.4    0.0    2.0   0  33 c2t5000CCA369C46666d0
  427.7  253.9 10232.6 10017.2  0.0  2.0    0.0    2.9   1  41 c2t5000CCA369C4498Cd0
    5.2  371.7  193.2 14771.4  0.0  2.5    0.0    6.5   0  41 c2t5000CCA369C46850d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    1.0   0   0 c2t5000CCA369C44AFFd0 (this is my  spare)

Why is the disk activity so high on the last 3 active drives in my pool? They are at between 33 and 41 %b and have much more reads and writes.
Can someone please shed some light on why this may be?