OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

@Gea:

Question regarding permissions because I haven't set something up properly even after 2 years (shame on me).

First off, let's just reset everything. I should probably start with a reset ACL because everything is screwed up after 2 years of stupidity. Do I reset to "Modify?" What should be left?

I want to setup 2 folders:

1) Media - read only access w/o login, but write access w/ login like root or say my username "dmo580"

2) Public - read only access w/o login, write access with a generic username like "user"

For Media: Would the best way to go about this to create a ZFS Filesystem that has guest access? I believe my current state does not have guest access. Is there any way to switch over, or would I have to create something brand new?

I assume the next steps are to set everyone@ to have read access only, and owner@ and root to have full access.

For Public:

Same thing regarding guest access?

Also a general question regarding trivial access like Owner@, how would you define who's an owner?
 
Last edited:
Regardless of using e1000 or vmxnet3 adapter on openindiana VM, I get disconnected the minute I try to start a VM on the NFS share (from openindiana to ESXi).

I've change the maxqueuedepth to 64. I've tested with and without jumbo frames (I have a separate vswitch for NFS)

The problem persists. ESXi event log shows device or file system with identifier xxxx-xxxx has entered the all paths down state

4 virtual cores
24Gb ram
mirrored rpool on 15k sas drives
all vdevs are 2 drive mirrors consisting of Hitachi 4tb drives.
 
@Gea:

Question regarding permissions because I haven't set something up properly even after 2 years (shame on me).

First off, let's just reset everything. I should probably start with a reset ACL because everything is screwed up after 2 years of stupidity. Do I reset to "Modify?" What should be left?

I want to setup 2 folders:

1) Media - read only access w/o login, but write access w/ login like root or say my username "dmo580"

2) Public - read only access w/o login, write access with a generic username like "user"

For Media: Would the best way to go about this to create a ZFS Filesystem that has guest access? I believe my current state does not have guest access. Is there any way to switch over, or would I have to create something brand new?

I assume the next steps are to set everyone@ to have read access only, and owner@ and root to have full access.

For Public:

Same thing regarding guest access?

Also a general question regarding trivial access like Owner@, how would you define who's an owner?

- You cannot set rules with/without login
If you enable guest, there is no login needed.

-owner is the account that creates a file.
You can modify with chown at cli
 
Had a strange issue with OmniOS/Napp-it. I had a accidental power failure (unplugged the wrong cord) that took down my ESXi setup. Upon restarting everything I could not get to Napp-it, I logged in through the console and got a error message (stupid me didn't screen-cap it). Rebooted the server and still nothing. Rebooted a 3rd time and it was back but I needed to reset the admin passwords and re-import my pools. Any idea what would cause this?

I had a similar issue on a machine that lost all of it's iSCSI shares (OmniOS/Napp-it Stable) when I took it from the office here to the data center.
 
Had a strange issue with OmniOS/Napp-it. I had a accidental power failure (unplugged the wrong cord) that took down my ESXi setup. Upon restarting everything I could not get to Napp-it, I logged in through the console and got a error message (stupid me didn't screen-cap it). Rebooted the server and still nothing. Rebooted a 3rd time and it was back but I needed to reset the admin passwords and re-import my pools. Any idea what would cause this?

I had a similar issue on a machine that lost all of it's iSCSI shares (OmniOS/Napp-it Stable) when I took it from the office here to the data center.

Sounds like you booted into a former data state/ boot environment (BE).

If you install napp-it via wget you are asked to reboot afterwards to boot into the newly
created and activated BE. If you do not reboot, all modifications that you have done
afterwards are lost on next reboot

If you have done some modifications after install that you want to preserve,
you can create a new BE at any time, activate and reboot.

If you only want to save a state, create a BE and do not activate.
If needed, you can then select this BE on startup.
 
- You cannot set rules with/without login
If you enable guest, there is no login needed.

But is it possible to set:

- No login needed to read/execute
- Login required to write?

I think I got it setup where I turned ON guest access but set trivial acl everyone@ to "read." therefore, unless I login as root for example, i can't copy files over. Would that be correct?
 
But is it possible to set:

- No login needed to read/execute
- Login required to write?

I think I got it setup where I turned ON guest access but set trivial acl everyone@ to "read." therefore, unless I login as root for example, i can't copy files over. Would that be correct?

Not possible with Solaris cifs (afaik) -
if you enable guest you are never asked for user/pw
while you need it always wihout guestaccess enabled.
 
Sounds like you booted into a former data state/ boot environment (BE).

If you install napp-it via wget you are asked to reboot afterwards to boot into the newly
created and activated BE. If you do not reboot, all modifications that you have done
afterwards are lost on next reboot

If you have done some modifications after install that you want to preserve,
you can create a new BE at any time, activate and reboot.

If you only want to save a state, create a BE and do not activate.
If needed, you can then select this BE on startup.

I think your are exactly right. So far I have had no more issues.
 
But is it possible to set:

- No login needed to read/execute
- Login required to write?

I think I got it setup where I turned ON guest access but set trivial acl everyone@ to "read." therefore, unless I login as root for example, i can't copy files over. Would that be correct?

http://i.imgur.com/ngj69bF.png

That's how I have my permissions setup. When I visit my server through Windows, it lets me browse that folder with no login, but I can't delete it. However if I login with root, I can then get write access? I thought that's what I was describing or did I do something really weird?

I also thought this guy was setting up a guest only access and got it to work?

http://forums.servethehome.com/sola...ting-up-configuring-smb-sifs-shares-napp.html
 
http://i.imgur.com/ngj69bF.png

That's how I have my permissions setup. When I visit my server through Windows, it lets me browse that folder with no login, but I can't delete it. However if I login with root, I can then get write access? I thought that's what I was describing or did I do something really weird?

I also thought this guy was setting up a guest only access and got it to work?

http://forums.servethehome.com/sola...ting-up-configuring-smb-sifs-shares-napp.html

You can access without login if your have a Solaris user with the same name and PW that you use on Windows.
 
Hi,
I have a little problem:
08f281f1-be19-41f9-83b2-f4312240a41b.jpg

Config is :
X9SRH-7TF with LSI2308 successfully flashed to IT mode, i also removed the "sas2flash.efi -b mptsas2.rom" line in the SMC2308T.NSH file to avoid long boot delay.
OS is OmniOS + napp-it

I plugged 10*5K4000 on it, 8 on the LSI2308 and 2 on the mobo SATA ports.
I can't see the drive connected, only two of them. (Probably the mobo ones).
 
Hi,

I plugged 10*5K4000 on it, 8 on the LSI2308 and 2 on the mobo SATA ports.
I can't see the drive connected, only two of them. (Probably the mobo ones).

This can happen if you have enabled partition support without vaild partitions on your disks.

Try:
- disable partitions support (menu disks - partitions) or
- initialize disks (menu disks-initialize)

try also:
delete disk buffer (menu disks - delete disk buffer)

or enter format at cli and check if the disks are basically available at os level
(cancel command after disk listing with ctrl-c)
 
Thanks!
I tried this but i still see only two drives.
Also there is only two drives blinking at boot.
 
Thanks!
I tried this but i still see only two drives.
Also there is only two drives blinking at boot.

If the disks are even missing with the format command, i suppose you use a wrong reverse sas cable (used to connect onboard sata to a sas backplane)
 
i see, X9SRH-7TF with LSI2308 over sata.
- should work without problem if you connect disks directly via sata (i use the same board).
Only with SAS backplanes you need a reverse sas cable.
 
Yeah, during boot only 2/5 hdds leds on the CSE-M35T are blinking, so the backplane should be ok.
I don't know where the problem is, the LSI2308 has been flashed to IT without problem.
Perhaps it need the mptsas2.rom to work?
 
You can access without login if your have a Solaris user with the same name and PW that you use on Windows.

I see what you mean, but what I'm saying is I setup a "test" folder with Guest login, meaning I should NOT have to login.

However in the ACL Extension, I made sure "Everyone@" is set to read only, and I added root to have full permissions. Therefore, when I try to access this "test" folder, it seems to allow me to view everything without login but if I want write access, I need to login as Root. This is exactly what I want, but you're saying that's not possible? Did I setup something incorrectly?
 
I tried to reflash the LSI2308 with full firmware+BIOS this time.
In the LSI BIOS menu (using CTRL+C) i can see the eight hdds! (and all the leds are blinking at least one time.)
But in napp-it i can't see them, i tried again to disable partition support, initialize disks, delete disk buffer and enter the format command, but i only see two hdds.
 
Is the firmware version important?
because i used a more recent one (PH16.0.1-IT).

Are your disks connected directly via Sata or via backplane?
If so try to connect directly.

Can you try another disk at the same port
I have had a problem with WD Raptors 330GB where some disks were not detected with LSI 2308 but ok with LSI 1068
 
All the disks are in racks, but only 2/10 of them are visible through napp-it.
I tried connecting another disk (with another sata cable) directly to a LSI2308 port, the disk is still invisible by napp-it.
 
All the disks are in racks, but only 2/10 of them are visible through napp-it.
I tried connecting another disk (with another sata cable) directly to a LSI2308 port, the disk is still invisible by napp-it.

If you cannot see any disks not even with CLI command format then the controller is damaged or something went wrong with flashing.
 
But the disks are all visible though the controller's BIOS (by pressing CTRL-C at boot).:confused:

Could it be the firmware version incompatible? (i used the PH16.0.1-IT instead of PH15-IT)

I also saw something about invisible disks in the X9SRH-7F LSI2308 IT flash topic :
Yeah, the controller is in IT/HBA mode and everything is fine, right?
Well, not if you're running an all-in-one system with OpenIndiana. :eek: I am running OI151a7 and it seems that the developers have not included the drivers for the LSI2308 yet as none of the drives connected to the LSI2308 was visible in OI.
Some more digging and it turns out that the mpt_sas driver is simply not associated with the chips device ID.

The following command (enter as root) does the trick of assigning the mpt_sas driver to the LSI2308:
Code:
update_drv -a –i "pci15d9,69l" mpt_sas

Now all connected drives show up fine and I can import the pool that was originally connected to the LSI 1068E on the original board (see 1st post).

I tried the command without success, maybe there is something to modify?
 
Gea,

I was troubleshooting a very strange problem today. I am using the active directory integration on my AIO and using it as a fileserver in domain mode. It works great and speeds are wonderful! However today, some users complained of slow connections to the shared drives. Sure enough, i could only transfer a 4gb ISO from the SAN to workstations at under 500 K/s. However other workstations chugged along at 90MB/s all day! I tried swapping cables, nics, switches to no avail. Finally I tried changing the workstation IP to be in the same subnet as the SAN and full transfer speeds returned! Our environment is "supernetted", so our subnet mask is 255.255.252.0 and we have 4 blocks of IPs. (ex. 192.168.140.0-192.168.143.254). I triple checked the subnet mask in all servers/workstations and it is entered correctly. The issue happens whether i use VMXNET3 or E1000g nics on the san vm. If i ping the san during file transfer, it will come back as follows:
Code:
Reply from 192.168.140.91: bytes=32 time<1ms TTL=255
Reply from 192.168.140.91: bytes=32 time=2033ms TTL=255
Reply from 192.168.140.91: bytes=32 time<1ms TTL=255
Reply from 192.168.140.91: bytes=32 time<1ms TTL=255
Reply from 192.168.140.91: bytes=32 time=2372ms TTL=255
Reply from 192.168.140.91: bytes=32 time<1ms TTL=255
Reply from 192.168.140.91: bytes=32 time<1ms TTL=255
Reply from 192.168.140.91: bytes=32 time=1365ms TTL=255
Reply from 192.168.140.91: bytes=32 time<1ms TTL=255
Reply from 192.168.140.91: bytes=32 time<1ms TTL=255
Reply from 192.168.140.91: bytes=32 time=2364ms TTL=255
Reply from 192.168.140.91: bytes=32 time<1ms TTL=255
Reply from 192.168.140.91: bytes=32 time<1ms TTL=255
Reply from 192.168.140.91: bytes=32 time<1ms TTL=255
Reply from 192.168.140.91: bytes=32 time=1364ms TTL=255
Reply from 192.168.140.91: bytes=32 time<1ms TTL=255
Reply from 192.168.140.91: bytes=32 time=2370ms TTL=255
Reply from 192.168.140.91: bytes=32 time<1ms TTL=255
Reply from 192.168.140.91: bytes=32 time<1ms TTL=255
Reply from 192.168.140.91: bytes=32 time=2371ms TTL=255
Reply from 192.168.140.91: bytes=32 time<1ms TTL=255
Reply from 192.168.140.91: bytes=32 time<1ms TTL=255

I would reboot the san, but i am worried i will loose all my "ephemeral" permissions on the shared folders. From what i gather these need to be recreated manually every reboot? Anyways, i was beginning to look at LSO or TCP offloading settings on the VMXNET3 driver, but none of my other boxes are exhibiting this behavior! Any insight would be greatly appreciated!
 
But the disks are all visible though the controller's BIOS (by pressing CTRL-C at boot).:confused:

Could it be the firmware version incompatible? (i used the PH16.0.1-IT instead of PH15-IT)

I also saw something about invisible disks in the X9SRH-7F LSI2308 IT flash topic :


I tried the command without success, maybe there is something to modify?
It does sound similar to the problem I was having on OpenIndiana. A quick check on whether this is a HW (cables, etc.) or SW problem, would be to do a quick install of another known-good OS temporarily. Install Windows as the main OS, install the LSI drivers and see if the eight drives show up correctly.
 
chune, I have seen odd performance anomalies when using the vmxnet3 vnic under OI and similar guests. Not consistently, but often enough that I won't use it (I stick with e1000). Long delays, bursts, etc...
 
chune, I have seen odd performance anomalies when using the vmxnet3 vnic under OI and similar guests. Not consistently, but often enough that I won't use it (I stick with e1000). Long delays, bursts, etc...

In esxi 5.5 the problem actually gets reversed and the e1000g nic is buggy out of the box while the vmxnet3 nic performs great. Gea makes note of this in his main page, instructing you to disable tcp offloading on the e1000 nic- however this VM was setup before i had been made aware of the issue. Anyways, on to the fix:

I was fortunate enough to have a CCIE staying with me and he was able to quickly drill down and see it was an ARP issue. In my case i had a total of three nics attached to my SAN VM, one for internal SAN traffic and the other two attached to the VM network (e1000g for mgmt and vmxnet3 for fileserver traffic). Apparently attaching two nics on one machine to the same network is bad practice and can lead to undesired behavior (packets coming in one interface and out another). Even though i specified the IP of the vmxnet3 adapter when initiating a SMB file transfer, a wireshark analysis revealed that the traffic was being returned through the undesired e1000g adapter. I will be disabling/removing the e1000g nic off hours and allowing the ARP to clear itself out. Hopefully this helps anyone else in the same boat!

As far as why the problem went away when changing the workstation IP to the different subnet was pure chance that it placed the ARP entry for the vmxnet3 adapter higher up on the SAN vm and it returned traffic from the proper interface.

So for another question, does anybody know if AD permissions need to be reset on ZFS CIFS shares after a reboot? Or is the SID/guid the only "ephemeral" part and it just generates a new one upon reboot?
 
Last edited:
chune, I have seen odd performance anomalies when using the vmxnet3 vnic under OI and similar guests. Not consistently, but often enough that I won't use it (I stick with e1000). Long delays, bursts, etc...
I suggest reading this excellent article about slow performance with vmxnet3 under solaris variants. I followed the suggestions in the blog and improved my AIO network performance substantially. Here is the link : Improving VM to VM network throughput on an ESXi platform. Note that I am running OI 151a7 (actually upgraded today to 151a8).

The main problem turned out to be that deep in the vmware-tools install script, they have an error that makes it use the vmxnet3 driver for Solaris 10 instead of the one for Solaris 11.

BTW, in the release notes for OI 151a8 they say:
* Include BETA vmxnet3s driver

Does anyone have any comments about this driver and whether it is better than the vmtools one?
 
Last edited:
I discovered after updating my OI 151a7 system to oi_151a8 today that the locate command is gone. A bit of googling shows that it was part of the now obsolete "slocate" package. The replacement package "mlocate" has not been released for Solaris and variants. Apparently the source will not compile properly under Solaris 11 variants. I rely on locate and having to use the old find command just won't cut it for me. So I am strongly considering going back to oi_151a7. Does OmniOS have locate?
 
Just run find(1) in a cronjob and grep(1) the results if needed. What are you running a UNIX for if you can't duplicate a simple task like locate(1)? :)
 
Just run find(1) in a cronjob and grep(1) the results if needed. What are you running a UNIX for if you can't duplicate a simple task like locate(1)? :)
You poor man's locate would sort of work, but the real locate has a database which produces the results very fast; and the updatedb command has many options to include/exclude directories from the search.
 
Yeah, locate is faster, but do you use it often enough that a slower workaround is worse than not having all the other new features you get with the update?

Including/excluding directories is a matter of providing proper options to find in the cronjob.
 
Hi all,

Can someone shed light on how to treat permission when mounting a CIFS share from a Ubuntu Server to my OmniOS Napp-it server?

Everytime I try to mount the CIFS share with a SMB user created on Napp-it, I'm not getting the same access rights as I do on fx. a Windows Server. It seems like Ubuntu is not using the SMB credentials, but in stead the unix permissions?

Working with the same share on a Windows Server, everything is working exactly as expected, but on Linux it seems to use local permissions or something.

Can anyone assist?

Thanks
Best regards
Jim
 
All the disks are in racks, but only 2/10 of them are visible through napp-it.
I tried connecting another disk (with another sata cable) directly to a LSI2308 port, the disk is still invisible by napp-it.

Have you tried removing partitions on the drives?

Eg completely wipe?

People have had troubles where the drives have been used before.... And are partitioned as GPT

Change them back to MBR and try again

.
 
I got a ignorant question.
What windows tool/dialog can I use to set permissions on a share/folder? I can't figure out how to connect/mount as (solaris user) root. I don't remember how I did it before. I don't think I ever set up an id map.
Storage server connected to AD domain.

edit
go find a computer with no mapped network drives and map the share as omnios\root
 
Last edited:
I got a ignorant question.
What windows tool/dialog can I use to set permissions on a share/folder? I can't figure out how to connect/mount as (solaris user) root. I don't remember how I did it before. I don't think I ever set up an id map.
Storage server connected to AD domain.

edit
go find a computer with no mapped network drives and map the share as omnios\root

Connect from any Windows computer as root (set root pw with passwd).
You are then able to set permissions (not on all Windows versions)

or
idmap any AD user (ex domainadmin) to root and connect from a Computer that is AD member
as domainadmin or domain\domainadmin if the computer is not an AD member
 
Back
Top