OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

Thanks to siamsquare for the screenshot, I have now disabled boot support of the card.

I have also removed the HW RAID and will go for the ZFS RAID - I am curious how it will behave.

Thanks a million times for the tip using LPMI virtual CD support - it worked flawlessly, I was able to run the LiveCD and will now start the OS installation.

You guys are great! :D
 
Just for amusement, try the following: create a raid1 via HW. Create a zfs pool on it and run a disk benchmark. Delete the zfs pool. Delete the HW mirror. Create a mirrored zfs pool. Rpeat the bench. See if the read performance is better.
 
Gea,

I've got a fairly large OpenIndiana box (72 - 2.5" drives) and every time i change pages on NAPP-IT it seems to scan every drive, taking a good 30+ seconds to change pages. Not only that, it seems to take (at least) SMB to a grinding halt while changing pages in NAPP-IT.

Curious if you've seen that, or have suggestions on improving that?

Thanks!
 
Just got Napp-It installed and I'm really happy with it, setup was so much easier than without using it!
 
I would love it if somebody would be willing to get on an IRC chat with me at some point and help me set permissions for samba shared folders. RIght now i've been having to use root to log in for everybody in my house. would rather if possible it log in using the same username password combo used to log into windows so that users don't have to enter anything at all(kinda like WHS does) I would really like to get this solved because users can see everything in another specific folder.
 
Gea,

I've got a fairly large OpenIndiana box (72 - 2.5" drives) and every time i change pages on NAPP-IT it seems to scan every drive, taking a good 30+ seconds to change pages. Not only that, it seems to take (at least) SMB to a grinding halt while changing pages in NAPP-IT.

Curious if you've seen that, or have suggestions on improving that?

Thanks!

i am working on that

napp-it is originally a pure web cgi-application what means, it must reread disk
and zfs config on every page. the more functionality i build in, the slower it becomes.

i am currently on optimizng speed by reading only needed information (helps a little)
and with background agents doing tasks in background.

At the moment, only optimation is to avoid slow system disks and to enable ZFS
buffering in top-menu to buffer last readed zfs config.
 
I would love it if somebody would be willing to get on an IRC chat with me at some point and help me set permissions for samba shared folders. RIght now i've been having to use root to log in for everybody in my house. would rather if possible it log in using the same username password combo used to log into windows so that users don't have to enter anything at all(kinda like WHS does) I would really like to get this solved because users can see everything in another specific folder.

First, i suppose you use the Kernel based CIFS/SMB server, not SAMBA

it should be quite easy with napp-it if you
- create your needed users on OpenIndiana
- verify that you have not any ID-mappings in workgroup mode
(check with menu services - smb - mapping)
- set needed permissions with napp-it menu extensions - acl
or with some Windows versions when connected as root.

best is to create a ZFS filesystem for each user and allow only this user or
use one ZFS filesystem with read permission for everyone@ for this folder only
(inheritance off) and create regular subfolders with ACL permissions for this folder.

Be aware: SMB is like Windows - ACL only !
 
hi gea


i think i messed up my smb group, now under 'user' menu, i can't see administrator group anymore.

failed to find An error occurred while retrieving group data.
(invalid name)
failed to find Check the system log for more information.
(invalid name)
 
Ughhhhhh.

Hoping for a miracle here. I have an All-in-one ESXi 5 box with OpenIndiana ZFS (pretty much per _Gea's guide) and it's been working great, using CIFS for Windows and AFP for Mac (though I'm having some issues with permissions on the Macs I still need to iron out).

Anyway, as a relatively new Mac user I found to my horror tonight that when moving folders someplace where folders of the same name exist, the destination folders are completely replaced and not merged (as in all other modern end-user OS's). Yes it's my fault for a multitude of reasons. First of all, no backup, secondly it pops up a dialog asking if you want to replace the folder--I just assumed that it meant merge because coming from Windows that's been the default behavior for years and years). Anyway, i quickly realized my mistake and tried to undo the operation... it moved the folders back to their original location but the originals they replaced are gone.

So afterwards I say, OK now I get to use that nifty snapshot feature I've heard about. So I log into OI and find the time slider deal. Doh. No snapshots. Apparently, snapshots aren't turned on by default and I never did so.

OK, so keeping in mind that this *just* happened 20 minutes ago and I have not made any appreciable changes to the file system (furthermore the files that were "moved" were already in the same datastore--I was just relocating them to a different subfolder) is there *any* hope I can recover the deleted files somehow? If this were a regular RAID array I'd just run one of any number of file undeleting utilities... but after a few google searches I couldn't find any mention of how this can be accomplished with ZFS.

If it cannot then this was a very costly lesson to me. Not really monetary cost but probably 30 hours of work down the drain...

sigh... much appreciate any help or advice.
 
hi gea


i think i messed up my smb group, now under 'user' menu, i can't see administrator group anymore.

failed to find An error occurred while retrieving group data.
(invalid name)
failed to find Check the system log for more information.
(invalid name)


i would go back to last system snap. Even a new install and pool-import is done
within 30 min and therefor faster than error finding
 
Ughhhhhh.

Hoping for a miracle here. I have an All-in-one ESXi 5 box with OpenIndiana ZFS (pretty much per _Gea's guide) and it's been working great, using CIFS for Windows and AFP for Mac (though I'm having some issues with permissions on the Macs I still need to iron out).

Anyway, as a relatively new Mac user I found to my horror tonight that when moving folders someplace where folders of the same name exist, the destination folders are completely replaced and not merged (as in all other modern end-user OS's). Yes it's my fault for a multitude of reasons. First of all, no backup, secondly it pops up a dialog asking if you want to replace the folder--I just assumed that it meant merge because coming from Windows that's been the default behavior for years and years). Anyway, i quickly realized my mistake and tried to undo the operation... it moved the folders back to their original location but the originals they replaced are gone.

So afterwards I say, OK now I get to use that nifty snapshot feature I've heard about. So I log into OI and find the time slider deal. Doh. No snapshots. Apparently, snapshots aren't turned on by default and I never did so.

OK, so keeping in mind that this *just* happened 20 minutes ago and I have not made any appreciable changes to the file system (furthermore the files that were "moved" were already in the same datastore--I was just relocating them to a different subfolder) is there *any* hope I can recover the deleted files somehow? If this were a regular RAID array I'd just run one of any number of file undeleting utilities... but after a few google searches I couldn't find any mention of how this can be accomplished with ZFS.

If it cannot then this was a very costly lesson to me. Not really monetary cost but probably 30 hours of work down the drain...

sigh... much appreciate any help or advice.

Usually your data is lost if you have no backups and no snaps.
I have not tried the SolarisInternal script - be careful, its quite old information.
I would not try without a backup.
 
I hard coded an ip in napp-it and it changed the network profile from "physical:nwam" to "physical:default". I later rebooted nexenta and the comstar iscsi target service did not start automatically. It said run "svcs -x svc:/network/iscsi/target:default" to determine why it is not online, it gave this result:

svc:/network/iscsi/target:default (iscsi target) State: offline since Sat Dec 17 22:33:24 2011
Reason: Service svc:/network/physical:default is not running because a method failed. See: http://sun.com/msg/SMF-8000-GE
Reason: Service svc:/network/physical:nwam is disabled. See: http://sun.com/msg/SMF-8000-GE See: itadm(1M) Impact: This service is not running.

Can't the iscsi target work when an ip is hard coded? My dhcp server cant do mac address reservation and the ip bounces around on reboots and i need it to stay the same that's why i hard coded it.

BUILD: napp-it nexenta3 appliance v. 0.6m nightly Dez.02.2011
 
Thanks s0rce and _Gea for the replies. I guess it's good that there's at least something one can try in such an instance but that procedure looks Hairy! Fortunately, what I lost can be re-created so I think I will just bite the bullet and do that rather than mess around trying to take my pools offline.

As usual, lessons are best learned the hard way and this is no exception! I learned that turning on ZFS snapshots is step #1 before you even load data on an array, and I learned that Mac OS still has a few areas in which it does not "just work" :)

Anyone reading this, learn from my mistake! Make sure you turn on snapshots for your pools!
 
...I'm at my next step, I would like to do aggregation on my two NIC's.
- can anyone having experience with that guide me or point me to a good guide? Since I am a Solaris noob, I would need a step-by-step guide...
- are there any known problems with napp-it in regards to aggregation? 'Cause that's gonna be my next step :D

Thanks guys, appreciate any help!

Cap'
 
Hi there,

I'm trying to go through and set up SMB sharing but am having a problem mounting from Ubuntu. So far I've followed the instructions and can connect to the SMB share from OS X. All the permissions are set up and I can create and edit files and they gain the owner and group attributes correctly.

However, from an Ubuntu VM, where I've followed the "Connecting using CIFS" guide (https://help.ubuntu.com/community/Samba/SambaClientGuide), when I mount the folder the owner and group names are shown as root:root and I don't have permission to create anything. How do I tell which user is being used to connect to the share from the Ubuntu world?

Thanks.
 
...I'm at my next step, I would like to do aggregation on my two NIC's.
- can anyone having experience with that guide me or point me to a good guide? Since I am a Solaris noob, I would need a step-by-step guide...
- are there any known problems with napp-it in regards to aggregation? 'Cause that's gonna be my next step :D

Thanks guys, appreciate any help!

Cap'

it is not supported by the napp-it web-UI.
You must do via CLI, google for Crossbow or look at Oracle docs.


do a system-snap prior testings so you can go back to a stable state without problem.
 
I've been experiencing a bunch of kernel panics with my Solaris 11 VM on ESXi 5. I don't know what is causing them. They're happening at mostly random intervals, usually with a couple of days in between (though just 8 minutes apart on one day). It does not appear to have any correlation to the server load, but strangely a bunch of them happen around a specific time (9:45 AM).

A typical entry in fmadm faulty looks like:

Code:
--------------- ------------------------------------  -------------- ---------
TIME            EVENT-ID                              MSG-ID         SEVERITY
--------------- ------------------------------------  -------------- ---------
Dec 18 09:45:55 77d310a4-23b6-63f0-a3ab-b0f88e923858  SUNOS-8000-KL  Major     

Host        : solaris1
Platform    : VMware-Virtual-Platform	Chassis_id  : VMware-56-4d-9a-6b-8a-d5-6c-08-96-f4-6b-ea-89-f4-97-ce
Product_sn  : 

Fault class : defect.sunos.kernel.panic
Affects     : sw:///:path=/var/crash/.77d310a4-23b6-63f0-a3ab-b0f88e923858
                  faulted but still in service
Problem in  : sw:///:path=/var/crash/.77d310a4-23b6-63f0-a3ab-b0f88e923858
                  faulted but still in service

Description : The system has rebooted after a kernel panic.

Response    : The failed system image was dumped to the dump device.  If
              savecore is enabled (see dumpadm(1M)) a copy of the dump will be
              written to the savecore directory /var/crash.

Impact      : There may be some performance impact while the panic is copied to
              the savecore directory.  Disk space usage by panics can be
              substantial.

Action      : Use 'fmadm faulty' to provide a more detailed view of this event.
              If savecore is not enabled then please take steps to preserve the
              crash image. Use 'fmdump -Vp -u
              77d310a4-23b6-63f0-a3ab-b0f88e923858' to view more panic detail.
              Please refer to the associated reference document at
              http://sun.com/msg/SUNOS-8000-KL for the latest service
              procedures and policies regarding this diagnosis.

Has anyone experienced similar issues or know how to solve the problem?
 
All right _Gea, thanks for the info.
Does that mean if I manage to get the aggregate going using CLI, napp-it does not have a problem with it?
Thanks,
Cap'
 
Hi there,

I'm trying to go through and set up SMB sharing but am having a problem mounting from Ubuntu. So far I've followed the instructions and can connect to the SMB share from OS X. All the permissions are set up and I can create and edit files and they gain the owner and group attributes correctly.

However, from an Ubuntu VM, where I've followed the "Connecting using CIFS" guide (https://help.ubuntu.com/community/Samba/SambaClientGuide), when I mount the folder the owner and group names are shown as root:root and I don't have permission to create anything. How do I tell which user is being used to connect to the share from the Ubuntu world?

Thanks.

Just to give some more information.

If I connect from OS X I am prompted for a username and password. When I list the share I see the owner:group as sjalloq:staff.

I cannot mount from Ubuntu unless I specify a username but it doesn't care if I specify a password or not. Once mounted all files/directories have owner:group of root:root. I don't have permission to create anything but using sudo creates files/directories with owner:group of sjalloq:staff.

Confused.
 
All right _Gea, thanks for the info.
Does that mean if I manage to get the aggregate going using CLI, napp-it does not have a problem with it?
Thanks,
Cap'

correct.
If you have enough Nics, left the management Nic untouched otherwise you must
set/ reset all from local console
 
Just to give some more information.

If I connect from OS X I am prompted for a username and password. When I list the share I see the owner:group as sjalloq:staff.

I cannot mount from Ubuntu unless I specify a username but it doesn't care if I specify a password or not. Once mounted all files/directories have owner:group of root:root. I don't have permission to create anything but using sudo creates files/directories with owner:group of sjalloq:staff.

Confused.

Do not forget: Solaris Kernel based CIFS server is ACL only, just like a real Windows server.
You must check against ACL not unix permissions
If you use napp-it, you may try the ACL extension. It is free for basic local user settings.
You can use it to check and set ACL
 
Do not forget: Solaris Kernel based CIFS server is ACL only, just like a real Windows server.
You must check against ACL not unix permissions
If you use napp-it, you may try the ACL extension. It is free for basic local user settings.
You can use it to check and set ACL

I appreciate that, but was I wrong in assuming that OS X and Ubuntu should behave in the same way?

My aim is to connect to the SMB share via Ubuntu and not have to run everything as root. It works correctly on OS X.
 
All right _Gea, thanks for the info.
Does that mean if I manage to get the aggregate going using CLI, napp-it does not have a problem with it?
Thanks,
Cap'

I used http://docs.oracle.com/cd/E19963-01/html/821-1458/fpjvl.html#scrolltoc to set up my LACP aggregation.

In napp-it you won't see it listed under the interfaces table, but it will be listed underneath the "Your current config"

Here are snippets from my config:

Code:
matt@solaris11box:~$ dladm show-link
LINK        CLASS     MTU    STATE    BRIDGE     OVER
e1000g2     phys      1500   unknown  --         --
e1000g3     phys      9216   up       --         --
e1000g0     phys      9216   up       --         --
e1000g1     phys      9216   up       --         --
aggr0       aggr      9216   up       --         e1000g0 e1000g1
You can see aggr0 is using e1000g0 and e1000g1

Code:
matt@solaris11box:~$ dladm show-aggr
LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER   FLAGS
aggr0           L4       auto                 active        short       -----

Code:
matt@solaris11box:~$ ipadm show-addr
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
aggr0/v4static    static   ok           192.168.2.1/24
e1000g3/v4static  static   ok           192.168.1.5/24
lo0/v6            static   ok           ::1/128

Code:
matt@solaris11box:~$ ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
aggr0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 9216 index 2
        inet 192.168.2.1 netmask ffffff00 broadcast 192.168.2.255
e1000g3: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 9216 index 3
        inet 192.168.1.5 netmask ffffff00 broadcast 192.168.1.255
 
I appreciate that, but was I wrong in assuming that OS X and Ubuntu should behave in the same way?

My aim is to connect to the SMB share via Ubuntu and not have to run everything as root. It works correctly on OS X.

It should not matter, wether your client is Wndows, Mac or Unix/Linux.
They connect as user a with pw b

You must have created such a user on Solaris and this user must have ACL permissions
on the shared folder and the share itself

(Beside client problems like we have on OSX 10.7)
 
It should not matter, wether your client is Wndows, Mac or Unix/Linux.
They connect as user a with pw b

You must have created such a user on Solaris and this user must have ACL permissions
on the shared folder and the share itself

(Beside client problems like we have on OSX 10.7)

Yes I have gone through what I think is the flow. I can successfully connect from OS X so it looks like it works. But as previously stated, it doesn't work from Ubuntu.
 
I figured I would ask here in case anyone knows but I'm trying to determine the total in/out data transfer used by my solaris machine.

I've found netstat -s returns a lot of data but I'm not sure which ones are important.

edit: found it, kstat -p e1000g0:0:*:*bytes*
 
Last edited:
Thanks adi for the detailed information.
So if I read this right, you use e1000g3 as the Mgmt interface, as suggested by _gea, and the aggregate on a different subnet for the data traffic?
On the switch side, do you configure the LACP after the Solaris config? Sorry for all these noob questions...
Thanks and regards,
Cap'
 
Just tried doing the aggregate according to the Oracle Doc, but it doesn't come up. I am fine to create the aggregate, and it is also configured on the switch. My config is as follows right now:

Code:
root@nas1:~# dladm show-link
LINK                CLASS     MTU    STATE    OVER
net1                phys      1500   up       --
net0                phys      1500   up       --
net3                phys      1500   up       --
aggr0               aggr      1500   up       net0 net1

Code:
root@nas1:~# dladm show-aggr
LINK              POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER   FLAGS
aggr0             L4       auto                 off           short       -----

Code:
root@nas1:~# ipadm show-addr
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
net3/_a           static   ok           192.168.1.222/24
lo0/v6            static   ok           ::1/128

Code:
root@nas1:~# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
	inet 127.0.0.1 netmask ff000000 
net3: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 9
	inet 192.168.1.222 netmask ffffff00 broadcast 192.168.1.255
	ether 0:15:17:d3:56:2 
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
	inet6 ::1/128

When I try to give an IP address to the aggregate:
Code:
root@nas1:~# ipadm create-addr aggr0 -T static -a 192.168.1.224/24 data/v4
I get an error "invalid argument".
I then noticed that according to the documentation, there's no interface name after "create-addr", but I get the same result when I don't add this.

I have the feeling that I have an aggregate, but no corresponding interface that could be assigned an IP address, is that correct? How can I correct this?

Your help is greatly appreciated - thanks!

Best regards,
Cap'
 
Just tried doing the aggregate according to the Oracle Doc, but it doesn't come up. I am fine to create the aggregate, and it is also configured on the switch. My config is as follows right now:

Code:
root@nas1:~# dladm show-link
LINK                CLASS     MTU    STATE    OVER
net1                phys      1500   up       --
net0                phys      1500   up       --
net3                phys      1500   up       --
aggr0               aggr      1500   up       net0 net1

Code:
root@nas1:~# dladm show-aggr
LINK              POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER   FLAGS
aggr0             L4       auto                 off           short       -----

Code:
root@nas1:~# ipadm show-addr
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
net3/_a           static   ok           192.168.1.222/24
lo0/v6            static   ok           ::1/128

Code:
root@nas1:~# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
	inet 127.0.0.1 netmask ff000000 
net3: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 9
	inet 192.168.1.222 netmask ffffff00 broadcast 192.168.1.255
	ether 0:15:17:d3:56:2 
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
	inet6 ::1/128

When I try to give an IP address to the aggregate:
Code:
root@nas1:~# ipadm create-addr aggr0 -T static -a 192.168.1.224/24 data/v4
I get an error "invalid argument".
I then noticed that according to the documentation, there's no interface name after "create-addr", but I get the same result when I don't add this.

I have the feeling that I have an aggregate, but no corresponding interface that could be assigned an IP address, is that correct? How can I correct this?

Your help is greatly appreciated - thanks!

Best regards,
Cap'
Plumb the aggregation:
Code:
ifconfig aggr0 plumb[/url]
See also [url=http://kalali.me/configuring-solaris-link-aggregation-ethernet-bonding/]complete directions[/url].
 
Thanks for the hint. I configured the interface as described
ifconfig aggr0 plumb 192.168.1.224/24
In the second I bring the interface up (ifconfig aggr0 up), I loose all network connectivity to the host, also to the management interface on 192.168.1.222. I then have to bring the interface down again on the console to regain access.
I have a Cisco Switch SG200-18 that should support LACP. I configured a LAG1 for Solaris with the two interfaces concerned. However, the LAG is always shown there as "down".

Any more ideas? Any help is greatly appreciated!
Thanks,
Cap'
 
Thanks for the hint. I configured the interface as described

In the second I bring the interface up (ifconfig aggr0 up), I loose all network connectivity to the host, also to the management interface on 192.168.1.222. I then have to bring the interface down again on the console to regain access.
I have a Cisco Switch SG200-18 that should support LACP. I configured a LAG1 for Solaris with the two interfaces concerned. However, the LAG is always shown there as "down".

Any more ideas? Any help is greatly appreciated!
Thanks,
Cap'

Did you remember to turn off NWAM? Network auto magic will really screw with you here, quietly un-doing many of your configuration changes right after you make each one. If you are going to set up non-standard network configs you need to turn off NWAM.

svcadm disable nwam
svcadm enable network/physical:default
 
Nope, did not think about that. I have now turned off NWAM and retried, but no luck.
I still cannot see the aggregate as an <addrobj> and hence cannot give it an IP...
Any more?
 
What are your exact steps? Were they these?

- turn off NWAM
- Create the aggregate
- plumb the aggregate
- activate it with an IP
 
Nope, did not think about that. I have now turned off NWAM and retried, but no luck.
I still cannot see the aggregate as an <addrobj> and hence cannot give it an IP...
Any more?

Depending on your switch port configuration

May want to try this
dladm modify-aggr -L active aggr0
 
@dilidolo, you're great. Thanks a lot, this has immediately solved my problem.

So summarizing, what I had to do:
1. turn off NWAM
Code:
svcadm disable nwam
svcadm enable network/physical:default
2. remove config from the adapters, if there is any (check with dladm show-link)
Code:
ipadm delete-if <interface>
3. have the switch configured to form an LACP on the two adapters
4. create the aggregate
Code:
dladm create-aggr [B]-L active[/B] -l net0 -l net1 aggr0
In my config, it was necessary to make the aggregate in Active mode - I honestly don't know if this is always the case.
5. plumb the aggregate
Code:
ifconfig aggr0 plumb 192.168.1.224/24 up

I think that's about it, I hope I have not forgotten anything. Last thing now is that the plumb is gone after a reboot. Does anyone know where I have to add this so that it remains persistent?

To all of you - thanks a lot for your help! This forum rocks!
 
Back
Top