OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

Still reports
Code:
echo ::sd_state |mdb -k | grep phy_blocksize
    un_phy_blocksize = 0x200
    un_phy_blocksize = 0x200
    un_phy_blocksize = 0x200
    un_phy_blocksize = 0x200
    un_phy_blocksize = 0x200
    un_phy_blocksize = 0x200
    un_phy_blocksize = 0x200
    un_phy_blocksize = 0x200

No Idea unfortunately.

Our padding certainly lines up (and mine is working):
Code:
sd-config-list = "ATA     WDC WD20EARS-22M", "physical-block-size:4096";
sd-config-list = "ATA     SAMSUNG HD204UI ", "physical-block-size:4096";

My Drives are in a JBOD box with a ck23601 expander via a pass thru m1015 (IT Mode) HBA on OI on top of ESXi5.1.

One question:

I manually edited my sd file via nano and I was logged in as root at the time.

My sd-config-list entry was put at the bottom of the sd file after everything else that was pre-existing.

Are you trying as an admin user?

Could be worth a try as root?
 
No Idea unfortunately.

Our padding certainly lines up (and mine is working):
Code:
sd-config-list = "ATA     WDC WD20EARS-22M", "physical-block-size:4096";
sd-config-list = "ATA     SAMSUNG HD204UI ", "physical-block-size:4096";

My Drives are in a JBOD box with a ck23601 expander via a pass thru m1015 (IT Mode) HBA on OI on top of ESXi5.1.

One question:

I manually edited my sd file via nano and I was logged in as root at the time.

My sd-config-list entry was put at the bottom of the sd file after everything else that was pre-existing.

Are you trying as an admin user?

Could be worth a try as root?

definitely doing it via root, have tried all sorts of padding combos / reboot -p / shutdown etc.

The sd-config-list line is definitely at the bottom as well.

I'm also OI 151a7 on top of ESXi 5.1 with my LSI card in pass through.

Strange thing is, the first time I did this in Solaris Express 11.1 - it worked fine with that padding.
 
definitely doing it via root, have tried all sorts of padding combos / reboot -p / shutdown etc.

The sd-config-list line is definitely at the bottom as well.

I'm also OI 151a7 on top of ESXi 5.1 with my LSI card in pass through.

Strange thing is, the first time I did this in Solaris Express 11.1 - it worked fine with that padding.

Can you try this and see if it will do a match (just a stab in the dark unfortunately):
Code:
sd-config-list = "ATA     WDC             ", "physical-block-size:4096";
sd-config-list = "ATA     SAMSUNG HD204UI ", "physical-block-size:4096";

Having one space short did not work for me previously.
 
Last edited:
Doing double LAG is a bad idea. You have high chance that a ends up on the same link.

For vmxnet3 enabling performance mode on esxi helped me


I get 114Mb write and 105MB read on vmxnet3 with OI.

LAG is helpful when more than one client are accessing the server.
I do LAG, and very helpful when doing backup without degrade much for client(s) access.

are you sure 114MB write and 105MB read?
check with your OI, I think you can pull ethernet information from the system. you can create a script and do calculation on the fly. this is raw data :).
as I understand, Read should be higher rate than writing in general rule of thumb.

on myside, e1000 give me better performance data rate :D.
 
LAG is helpful when more than one client are accessing the server.
I do LAG, and very helpful when doing backup without degrade much for client(s) access.

are you sure 114MB write and 105MB read?
check with your OI, I think you can pull ethernet information from the system. you can create a script and do calculation on the fly. this is raw data :).
as I understand, Read should be higher rate than writing in general rule of thumb.

on myside, e1000 give me better performance data rate :D.

He is referring to Double LAG, if the poster is doing lag on the ESXi host and on the Guest that is bad.
 
Can you try this and see if it will do a match (just a stab in the dark unfortunately):
Code:
sd-config-list = "ATA     WDC             ", "physical-block-size:4096";
sd-config-list = "ATA     SAMSUNG HD204UI ", "physical-block-size:4096";

Having one space short did not work for me previously.

nope, unfortunately no luck with that either.

this is f$%#@!, I can't believe it will not work.

Does anyone else have an idea why i can't ashift=12 these WD drives on any illumos based OS?
 
Last edited:
LAG is helpful when more than one client are accessing the server.
I do LAG, and very helpful when doing backup without degrade much for client(s) access.

are you sure 114MB write and 105MB read?
check with your OI, I think you can pull ethernet information from the system. you can create a script and do calculation on the fly. this is raw data :).
as I understand, Read should be higher rate than writing in general rule of thumb.

on myside, e1000 give me better performance data rate :D.

As mentioned above, I am talking about double LAG.

The speed I am mentioning are the stat reported by the Win8 Filetransfer. I haven't fully figured out why my read are slower but I think it might be something with the "poor" driver from Realtek.
 
As mentioned above, I am talking about double LAG.

The speed I am mentioning are the stat reported by the Win8 Filetransfer. I haven't fully figured out why my read are slower but I think it might be something with the "poor" driver from Realtek.

m1abram, already mentioned for correcting me :)

double LAG is doable, but not recomended haha....

if you want to measure the raw data rate, check on OI directly.

I do not pay attention much on windows GUI data rate ( have windows 7 and windows 8), since windows seems to me , do not display rate correctly.

Usually I capture ethernet information data on the OS under /dev/sys live system ( just create a script to do calculation on the fly, and set the duration sampling time). do not know on Windows) to know the real raw data rate.

those are just my practical approach, everyone can be different.
 
@_Gea:

Is it possible to add the option of defining your own ACL Reset parameters?

I would like to reset all my ACL's with the following parameters(just example):
Code:
     0:group:vh31777:list_directory/read_data/add_file/write_data
         /add_subdirectory/append_data/read_xattr/write_xattr/execute
         /delete_child/read_attributes/write_attributes/delete/read_acl
         /write_acl/write_owner/synchronize:file_inherit/dir_inherit:allow
     1:group:vh31mod:list_directory/read_data/add_file/write_data
         /add_subdirectory/append_data/read_xattr/write_xattr/execute
         /delete_child/read_attributes/write_attributes/delete/read_acl
         /synchronize:file_inherit/dir_inherit:allow
     2:group:vh31rx:list_directory/read_data/read_xattr/execute
         /read_attributes/read_acl:file_inherit/dir_inherit:allow
     3:owner@:list_directory/read_data/add_file/write_data/add_subdirectory
         /append_data/read_xattr/write_xattr/execute/delete_child
         /read_attributes/write_attributes/delete/read_acl/write_acl
         /write_owner/synchronize:file_inherit/dir_inherit:allow

But if I enter then manually on the top level directory, they will not propagate. Only your "Reset ACL's" menu will propagate to subfolders and files. So an option to re-propagate top level directory ACL's to subfolders and files would be awesome.

Possible?

Thanks
Jim
 
nope, unfortunately no luck with that either.

this is f$%#@!, I can't believe it will not work.

Does anyone else have an idea why i can't ashift=12 these WD drives on any illumos based OS?

I suppose last resort approach is to create your pool under bootable ZFSGuru/Nas4free and export it and then import it into OI.

I had problems running freebsd with passthru under ESXi a while ago.
Posted here solution here:
http://hardforum.com/showthread.php?p=1038483037&highlight=#post1038483037

So I suggest just removing your ESXi drives and booting your ESX machine off a freebsd ZFSGuru/Nas4free stick, creating the pool/vdevs as Ashift 12 and then booting back into ESXi.

I also did not like what it did with my drives (the partition tables etc) which is why I moved to OI, but it seems you don't have many options left.
 
I suppose last resort approach is to create your pool under bootable ZFSGuru/Nas4free and export it and then import it into OI.

I had problems running freebsd with passthru under ESXi a while ago.
Posted here solution here:
http://hardforum.com/showthread.php?p=1038483037&highlight=#post1038483037

So I suggest just removing your ESXi drives and booting your ESX machine off a freebsd ZFSGuru/Nas4free stick, creating the pool/vdevs as Ashift 12 and then booting back into ESXi.

I also did not like what it did with my drives (the partition tables etc) which is why I moved to OI, but it seems you don't have many options left.

yeah that's what I was thinking until I booted of the ZFS Guru DVD and it wouldn't let me do anything until a DHCP address was received - come to think of it, my ports are tagged on the switch so i'll untag and try again.

Update:
Ran ZFSGuru and created a Ashift=12 pool and exported. Imported to OI via Napp-IT and good to see it still shows Ashift=12 haha.
It did however complain about my disks all being the same 'label' (GPT) rather then (GPT1 and 2 etc) - could this play part of why OI can't force ashift 12?
 
Last edited:
@_Gea:

Is it possible to add the option of defining your own ACL Reset parameters?

I would like to reset all my ACL's with the following parameters(just example):
Code:
     0:group:vh31777:list_directory/read_data/add_file/write_data
         /add_subdirectory/append_data/read_xattr/write_xattr/execute
         /delete_child/read_attributes/write_attributes/delete/read_acl
         /write_acl/write_owner/synchronize:file_inherit/dir_inherit:allow
     1:group:vh31mod:list_directory/read_data/add_file/write_data
         /add_subdirectory/append_data/read_xattr/write_xattr/execute
         /delete_child/read_attributes/write_attributes/delete/read_acl
         /synchronize:file_inherit/dir_inherit:allow
     2:group:vh31rx:list_directory/read_data/read_xattr/execute
         /read_attributes/read_acl:file_inherit/dir_inherit:allow
     3:owner@:list_directory/read_data/add_file/write_data/add_subdirectory
         /append_data/read_xattr/write_xattr/execute/delete_child
         /read_attributes/write_attributes/delete/read_acl/write_acl
         /write_owner/synchronize:file_inherit/dir_inherit:allow

But if I enter then manually on the top level directory, they will not propagate. Only your "Reset ACL's" menu will propagate to subfolders and files. So an option to re-propagate top level directory ACL's to subfolders and files would be awesome.

Possible?

Thanks
Jim

As a temporary solution I've created this script, basically it does the job.

Code:
/usr/bin/chmod -R A=owner@:list_directory/read_data/add_file/write_data/add_subdirectory/append_data/read_xattr/write_xattr/execute/delete_child/read_attributes/write_attributes/delete/read_acl/write_acl/write_owner/synchronize:file_inherit/dir_inherit:allow,group@:list_directory/read_data/add_file/write_data/add_subdirectory/append_data/read_xattr/write_xattr/execute/delete_child/read_attributes/write_attributes/delete/read_acl/synchronize:file_inherit/dir_inherit:allow,group:vh31777:list_directory/read_data/add_file/write_data/add_subdirectory/append_data/read_xattr/write_xattr/execute/delete_child/read_attributes/write_attributes/delete/read_acl/write_acl/write_owner/synchronize:file_inherit/dir_inherit:allow,group:vh31mod:list_directory/read_data/add_file/write_data/add_subdirectory/append_data/read_xattr/write_xattr/execute/delete_child/read_attributes/write_attributes/delete/read_acl/synchronize:file_inherit/dir_inherit:allow,group:vh31rx:list_directory/read_data/read_xattr/execute/read_attributes/read_acl:file_inherit/dir_inherit:allow,everyone@:list_directory/read_data/add_file/write_data/add_subdirectory/append_data/read_xattr/write_xattr/execute/delete_child/read_attributes/write_attributes/delete/read_acl/write_acl/write_owner/synchronize:file_inherit/dir_inherit:deny /share/
 
Hi all,

I am struggling with giving full windows pc access to a zfs folder with full permissions - ie no username/password on connection and full access/inheritance to new and existing data.

I have been reading a bit and have tried the following:

Confirmed devices are all on WORKGROUP
Ensure there were no ID Maps
In "ACL on SMB Shares" I have set @everyone to full_set with full access.
In the ZFS Folder, the dataset has guestok, e=full and full_set on smb-share-all.

Is there anything else I am missing?

With the above settings I can see the + next to the /dpool/datatank's unix permissions but this doesnt carry through to the sub folders.

Thankyou!

Paul

zfsfolder.png


smbacl.png
 
It did however complain about my disks all being the same 'label' (GPT) rather then (GPT1 and 2 etc) - could this play part of why OI can't force ashift 12?

Good to hear. :D

I think it did that with mine as well, way back, because the drives had pre-existing partitions that are not compatible. I used gparted to remove all partitions and then initialised them with ZFSGuru.
 
Hi all,

I am struggling with giving full windows pc access to a zfs folder with full permissions - ie no username/password on connection and full access/inheritance to new and existing data.

I have been reading a bit and have tried the following:

Confirmed devices are all on WORKGROUP
Ensure there were no ID Maps
In "ACL on SMB Shares" I have set @everyone to full_set with full access.
In the ZFS Folder, the dataset has guestok, e=full and full_set on smb-share-all.

Is there anything else I am missing?

With the above settings I can see the + next to the /dpool/datatank's unix permissions but this doesnt carry through to the sub folders.

Thankyou!

Paul

zfsfolder.png


smbacl.png


You are forgetting your "dpool" folder. Even though it's not a share, initial authentication polls that folder(at least that's what I'm seeing). Try this from a SSH session/console.

Clean current ACL:
Code:
/usr/bin/chmod -R A- /dpool/

Allow everyone to access dpool and dpool/datatank and set inheritance accordingly:
Code:
/usr/bin/chmod -R A=everyone@:list_directory/read_data/add_file/write_data/add_subdirectory/append_data/read_xattr/write_xattr/execute/delete_child/read_attributes/write_attributes/delete/read_acl/write_acl/write_owner/synchronize:file_inherit/dir_inherit:allow /dpool/
Remember to set guest ok via napp-it afterwards.
P.S Might be an idea to upgrade napp-it as well.

/Jim
 
You are forgetting your "dpool" folder. Even though it's not a share, initial authentication polls that folder(at least that's what I'm seeing). Try this from a SSH session/console.

Clean current ACL:
Code:
/usr/bin/chmod -R A- /dpool/

Allow everyone to access dpool and dpool/datatank and set inheritance accordingly:
Code:
/usr/bin/chmod -R A=everyone@:list_directory/read_data/add_file/write_data/add_subdirectory/append_data/read_xattr/write_xattr/execute/delete_child/read_attributes/write_attributes/delete/read_acl/write_acl/write_owner/synchronize:file_inherit/dir_inherit:allow /dpool/
Remember to set guest ok via napp-it afterwards.
P.S Might be an idea to upgrade napp-it as well.

/Jim

Thanks Jim,

Was working on something similar from the cli; i couldnt figure out why they kept dropping off until I checked my jobs and noticed a chmod 777 on this folder which was wiping out the ACLs.

Ive now set it at the command line with a
Code:
/usr/bin/chmod -R A+everyone@:full_set:fd:allow /dpool

And all is well :)

Thankyou
Paul

PS: Need to update, just not looking forward to re-creating my jobs/AFP and anything else so Ill hold out :)
 
So seems like my OI box is giving me a problem yet I don't know what it is.

I cannot access any of my shares... not from my linux boxes nor my windows boxes.... I checked all the status screens through napp-it to see if there are any errors but nothing.. no HDD errors, no jobs being run nothing.

It just happened out of the blue over night some time I suppose.

Any suggestions on where I should look before I reboot the box to see if it comes back to life?

EDIT: It seems the shares are still available... and browsing to the server via IP or host name works... but accessing the shares/server when going over the host name is extremely slow. If I browse the server/shares via IP address everything is quick. DNS checks out so I don't know what the issue could be. Also it only seems to be the SMB shares giving the issue, I can access the NFS shares either via host name or IP perfectly.

EDIT 2: A restart of the network services on the server seems to have fixed it. Strange.
 
Last edited:
I'm getting wildly different random read results when testing against different volumes in the same pool.

VOLUME 1
Test name Latency Avg iops Avg MBps cpu load
Max Throughput-100%Read 17.94 3282 102 6%
RealLife-60%Rand-65%Read 145.74 365 2 11% :confused:
Max Throughput-50%Read 10.17 5639 176 8%
Random-8k-70%Read 153.67 333 2 10% :eek:

This appears to have been the result of running:
svcadm disable network/physical:nwam​
without the corresponding
svcadm enable network/physical:default​

I'd be interested if anyone understands why that config mistake would produce such a narrowly limited effect. Or are the effects of that error kind of random and difficult to predict. I worked out that I'd left off the 2nd command because I kept having to re-plumb my aggr after restarts.
 
False positive.

Now the slow non-sequential performance is jumping around to all 3 volumes in the pool so at least it's consistent. I appear to be able to get as high as 95MBps in the non-sequential tests but frequently as low as 1MBps. Scrub is clean. I've turned off dedup. Arc REAL hit ratio is 92%. No ZIL. Statistics look good across the board.
 
OK - I think I understand. And if I've got this right I appreciate the fact that noone got on and called me a stupid noobie and i should do some reading before wasting peoples time with questions that have already been answered 1000 times.

My pool is Raid Z2 on a single 6 disk vdev. So the IOps cabability of the slowest disk in the pool is my upper limit on a random reads? But isn't there some magic that turns random writes into sequential writes? I don't think I'm seeing that in my 1-2MBps random test results.)

So where I'm seeing IOps around 10,000 on a random test, I must be hitting cache - I have 48Gb of ram on the box supplying ARC cache. The poor performance is moving from volume to volume as the cache optimizes itself to my tests?

So a ZIL mirror should help alot, right? I turned off sync and didn't see a performance increase. Should I have expected to?

thanks,
jb
 
Curious if anyone is running SMP on OpenIndiana or FreeBSD with the all-in-one? Will this work?
 
zpool set feature@lz4_compress=enabled <pool>
cannot set property for '<pool>': invalid feature 'lz4_compress'

anyone have any idea why i cant enable lz4? yes i replaced <pool> with my pool name.
 
zpool set feature@lz4_compress=enabled <pool>
cannot set property for '<pool>': invalid feature 'lz4_compress'

anyone have any idea why i cant enable lz4? yes i replaced <pool> with my pool name.

You need OmniOS bloody for LZ4
(OmniOS stable and OI have currently no LZ4 support - too new)
 
I'm coverting my z2 pool to a 3-vdev r10 pool to improve non-sequetial performance. But I have production backup data on two volumes that I don't want to destory. I have an empty pool on a second OI box Box2, so I thought I'd replicate my backup data to Box2, then destroy and create volumes on Box1 and replicate the data back to Box1.

Instead of using send receive to a remote host over ssh (zfs send -i scratchpool@snapshot1 scratchpool@snapshot2 |ssh id@host pfexec zfs recv slavepool) I mounted an iSCSI volume from Box2 on Box1 and then did a local replication. So 'zfs list' on Box1 knows about them but Box2 does not. How would I mount these volumes on Box2 now? I have the initial and an incremental snapshot of each.

thanks,
jb
 
nevermind, since I don't actually need them mounted on Box2, I think I can just destroy/create my Box1 pool and then clone from the snaps from the iscsi mount. Though I am academically curious how one might access the snaps on Box2 were it necessary ?
 
I just did a quick install of OmniOS Bleeding and the latest napp-it. For the life of me I cannot find how to install/enable the rsync server? It is not even listed in my available services. Thanks in advance for the help.
 
Anyone seen the problem where you cant set nbmand to off either via the gui or cli?

Via the gui I get the pop up but nothing happens, via the cli it accepts the comment but a zfs get all shows nothing has changed.

Any ideas?
 
Anyone seen the problem where you cant set nbmand to off either via the gui or cli?

Via the gui I get the pop up but nothing happens, via the cli it accepts the comment but a zfs get all shows nothing has changed.

Any ideas?

With napp-it its in the menu zfs filesystems/folders.
It requires an unmount/mount to become active
With current napp-it this is done automatically if the filesystem is not locked by a process.
 
Having trouble replacing a drive...
A year or two ago, I built a RAID-Z2 with 5 WD15EARS. One of them recently faulted, so I bought a WD15EZRX and tried a zpool replace. It said "devices have different sector alignment"

I started searching around and tried everything I could find. Can't get it to take. My pool says it's ashift=9 and this EZRX says it's 512 sector size, so I'm not even sure what the problem is.
Most of the search results I found were from awhile ago, 2011 and before.

Is there any chance upgrading OI will help? I'm on 151a.
 
Having trouble replacing a drive...
A year or two ago, I built a RAID-Z2 with 5 WD15EARS. One of them recently faulted, so I bought a WD15EZRX and tried a zpool replace. It said "devices have different sector alignment"

I started searching around and tried everything I could find. Can't get it to take. My pool says it's ashift=9 and this EZRX says it's 512 sector size, so I'm not even sure what the problem is.
Most of the search results I found were from awhile ago, 2011 and before.

Is there any chance upgrading OI will help? I'm on 151a.

This is a special problem of first generation 4k disks like WD EARS.
Although they are 4k disks, they lie about and offer themselves to the OS as 512B disks (physical and logical).

The problem now is that you cannot replace a 512Byte disk with ashift=9 with a modern 4k disks
that report the correct physical sector value. You need a 512Byte disk like old WD EARS or maybee a Hitachi 5k300
 
Currently running latest omnios bloody with napp-it v. 0.9a9

Has anyone been able to get gmail working for smtp?

When i try to install net-ssleay i get this error:
Creating Plan
pkg install: The following pattern(s) did not match any allowable packages. Try
using a different matching pattern, or refreshing publisher information:

net-ssleay
 
Currently running latest omnios bloody with napp-it v. 0.9a9

Has anyone been able to get gmail working for smtp?

When i try to install net-ssleay i get this error:
Creating Plan
pkg install: The following pattern(s) did not match any allowable packages. Try
using a different matching pattern, or refreshing publisher information:

net-ssleay

Its in the repo:
http://pkg.omniti.com/omniti-perl/en/index.shtml
other repos: http://omnios.omniti.com/wiki.php/Packaging

read:
http://omnios.omniti.com/wiki.php/GeneralAdministration#ConfigurePublishers

Configure Publishers #
List configured publishers: pkg publisher
Add a publisher (example): pkg set-publisher -g http://pkg.omniti.com/omniti-ms/ ms.omniti.com
Remove a publisher: pkg unset-publisher ms.omniti.com
 
Hi,
I have a problem with the configuration of xampp with napp-it on my openindiana server.

mysql won't start when i try to start all the services on napp-it. do i have anything to do ? I have never configured any webserver before so...
 
Hi,
I have a problem with the configuration of xampp with napp-it on my openindiana server.

mysql won't start when i try to start all the services on napp-it. do i have anything to do ? I have never configured any webserver before so...

MySQL on default XAMPP is not working on OI server.
It is ok with OI desktop/live and OmniOS
 
Am I the only one that still has lots of errors on omnios, spamming the console? It's virtually impossible to do anything via CLI :(
 
Am I the only one that still has lots of errors on omnios, spamming the console? It's virtually impossible to do anything via CLI :(

Every programm that is started from root or sudo outputs errors, warnings and control messages to the root console.
You cannot disable this but you can:

Use root as a role only.
-Login as another user and call admin tasks via su
-Remote login via putty and duplicate root console

Other option is to reduce messages. Most of them are:
- Warnings about missing hostname to ip resolving
If you add an entry in /etc/hosts for 127.0.0.1 hostname, this is solved

- Control messages whenever a programm is called via sudo
This is mostly due to admin apps like napp-it. You can switch them off in /etc/sudoers

If you use napp-it, you can solve the last two with updating to newest napp-it 0.9a9+,
reenter hostname (menu System -Network - Hostname) and a reboot

http://www.napp-it.org/downloads/changelog.html
 
Back
Top