OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

They are SATA - model ST31000340NS but they are supposed to be enterprise class. This was the barracuda ES line before they came up with the constellation line.

Yeah I have the SAS ES.2 version, hoping i do not run into the same issue one day :eek:
 
So when I found the bad drive, I deleted my pool and recreated it. I then removed the bad drive before I put any data on it (so I could still have a RAIDZ2 of 6 drives, currently running off 5). So if the bad drive was causing issues, that should stop. However even without adding in any SSDs for l2arc/zil, I am still having problems. Right now a simple read over iSCSI is sitting at 4.04 KB/s and has been for 20 minutes.
 
So when I found the bad drive, I deleted my pool and recreated it. I then removed the bad drive before I put any data on it (so I could still have a RAIDZ2 of 6 drives, currently running off 5). So if the bad drive was causing issues, that should stop. However even without adding in any SSDs for l2arc/zil, I am still having problems. Right now a simple read over iSCSI is sitting at 4.04 KB/s and has been for 20 minutes.

If you have just deleted and rebuilt your pool and are having problems and have not done all the work copying your data back in then I would suggest you quickly try splitting your drives up into temp smaller pools and test them quickly over iSCSI individually first. You could do 5 single disk pools (just for testing and not for use!). or maybe a couple of mirrors. or one mirror and a 3 disk RaidZ1 pool. Do all the mini pools have the same performance issues?

At least then you will know 100% if it is your disks or something else causing this.

Another option is look at finding a old disk your not using which is a different model/make and adding that as a single disk to compare performance with this as well. you could even add your l2arc SSD as a single disk pool and test that to compare with. If an SSD alone is slow then you have real issues (note expect slower write performance from just 1 SSD if you have sync write enabled because of extra ZIL writes)

Michael
 
Last edited:
So I have my OI+NAPP-IT attached to a domain.
I have a Folder mpool/media that has smb share and nfs on

The smb side of things seem to work well, I configured my user permissions on the share for my domain users.
ls -l of the mpool
drwxrwxrwx+ 7 2147483652 root 7 2012-10-18 20:23 media

Notice the owner uid?

If I ls -l the media folder:
ls: cannot access Music: Permission denied
ls: cannot access Shared Documents: Permission denied
ls: cannot access Videos: Permission denied
ls: cannot access Pictures: Permission denied
total 0
?????????? ? ? ? ? ? Music
?????????? ? ? ? ? ? Pictures

I get the same results from any machine that mounts the export via nfs. Any ideas?
Thanks.
 
So I have my OI+NAPP-IT attached to a domain.
I have a Folder mpool/media that has smb share and nfs on

The smb side of things seem to work well, I configured my user permissions on the share for my domain users.
ls -l of the mpool
drwxrwxrwx+ 7 2147483652 root 7 2012-10-18 20:23 media

Notice the owner uid?

The user id 2147483652 is a temporary mapping to the Windows (SID) user who creates the folder.
CIFS use ACL only and stores Windows SID as extended ZFS attributes to be fully Windows compatible


Your listing shows only Unix permissions, not the really used ACL (hides in the + after the listing)
NFS3 must care about these ACL settings but has no idea of a user.
So you must allow either everyone or nobody to give NFS access

or
your NFS client must have root permissions

Easiest way for NFS and CIFS
Code:
-  set ACL everyone@ to modify on the shared folder and enable inheritance
-  for current files and folders, you must reset ACL recursivly to this
-  do not touch Unix permissions when using CIFS, this will reset your ACL and inheritance settings - use always ACL
 
Hey Gea,
I recently made the jump from SE11 to OI 151A7 + NAPP-IT. My old setup used an ugly bit of python script to keep me posted on the servers status.
Now that I made the switch, I'd like to make use of the mailing jobs, NAPP-IT provides. Are there any templates that I could extend?
I want to implement FMA/SMF reports and temperature/rpm readouts via ipmitool / smartctl.
I tried tinkering with the action.pl in '/var/web-gui/data/napp-it/zfsos/15_jobs and data services/04_TLS Email/01_status' but my changes didn't seem to have any effect.
(I'm using TLS Mail with GMail, so I think this should be the appropriate file?)
Well, that's my question, now to a small bug report - free of charge!
I don't know if anyone has reported this yet, but the password form for the smtp password only stores 16 characters. This had me yelling at my terminal for a few minutes before I figured out that I could edit the config file. Perhaps you might want to fix that :)

-Frozen

//edit:
I've found another problem:
The status mail works just fine, but I don't receive any mails notifying me about the critical pool status...

My testpool:
http://twitpic.com/b5wxex
Alert job log:
http://twitpic.com/b5wxl7
 
Last edited:
Hey Gea,
I recently made the jump from SE11 to OI 151A7 + NAPP-IT. My old setup used an ugly bit of python script to keep me posted on the servers status.
Now that I made the switch, I'd like to make use of the mailing jobs, NAPP-IT provides. Are there any templates that I could extend?
I want to implement FMA/SMF reports and temperature/rpm readouts via ipmitool / smartctl.
I tried tinkering with the action.pl in '/var/web-gui/data/napp-it/zfsos/15_jobs and data services/04_TLS Email/01_status' but my changes didn't seem to have any effect.
(I'm using TLS Mail with GMail, so I think this should be the appropriate file?)
Well, that's my question, now to a small bug report - free of charge!
I don't know if anyone has reported this yet, but the password form for the smtp password only stores 16 characters. This had me yelling at my terminal for a few minutes before I figured out that I could edit the config file. Perhaps you might want to fix that :)

-Frozen

//edit:
I've found another problem:
The status mail works just fine, but I don't receive any mails notifying me about the critical pool status...

My testpool:
http://twitpic.com/b5wxex
Alert job log:
http://twitpic.com/b5wxl7

The files in '/var/web-gui/data/napp-it/zfsos/15_jobs and data services/* are templates for newly created jobs.
These jobs are stored in the folder '/var/web-gui/data/napp-it/_log/jobs.

If you want to modify a job, you must edit the jobfile, not the template.

About the alert-mails:
The job is started every 15 minutes to check the state but napp-it sends alerts only after an event and then once every day. (see jobscript if you want to edit)

More than 16 char for smtp?
I can change that in the next napp-it 0.9 release
 
Last edited:
Just to say up front, this all in one system totally rocks, and if I had not messed with it, it would still be fine. Thanks Gea.

A few weeks ago I got a bug to upgrade the memory in my server from 8gb to 16gb, I wanted to be-able to run more virtual machines with out affecting the critical parts of the server, pf-sense and OI+napp-it. Sounds good. no? I installed new ram and bumped ram for OI to 8gb from 6gb and rebooted. everything ran fine till yesterday I was moving some files around and deleting some files I no longer needed, I was doing this from within OI because explorer kept dropping the copy or just closing, but nautilus in OI was doing the same thing, so I rebooted...

When I upgraded the ram for OI I am fairly sure it automagicly increased the size of the swap file from 6gb to 8gb, and after upgading from OI 151a1 to 151a6 and only having 12gb to play with I was completely out of space, and OI will no longer boot.

I have very little experience with OI but i figure I will just make a new OI vm and import the pool, this actually worked great. The only issue I had with the install was copy past from the pdf had a character wrong (pdf wget -O - www.napp-it.org/nappit I perl) (web page wget -O - www.napp-it.org/nappit | perl) very small difference but it makes it so you cant copy and past from the pdf and make it work. Importing the pool was cake and only left me with one more hurdel, how to access my folders, it kept asking for username and password to access my folders from windows but the user/pass I had used before does not work. As a quick work around i just made some new folders and from within OI copied all files to new folders, and everything seems to work.

Sorry about this huge long story, but is there a better way to recover than what i found?
I found this link for upgrading disk for OI but i am not sure that is any faster than what I did. http://hardforum.com/showthread.php?t=1680337

Any idea about folder passwords? I was looking in Napp-it for a way to modify the folders I think I saw it on an older version but I could not find it on the new version.

thanks
rody
 
hello Rody

If you have bricked a napp-it/ OpenIndiana NAS, you must
- reinstall OI
- reinstall napp-it
- import your datapool
- re-create users

(done in about 30 min)

optionally: reset old ACL settings recursively + set new ACL as needed
(from Windows as user root or via napp-it and ACL extensions)

ps
- there are no folder passworrds. permissions are user permissions
- if you tell me the Page/URL I can fix a type problem.
 
wow all I had to do was add a user. lol
Thanks Gea.

in the pdf section 8.4 the link is
wget -O - www.napp-it.org/nappit I perl

on the web page it is
wget -O - www.napp-it.org/nappit | perl

the deference seems to be just this symbol | it is correct on the web page but in the pdf it is a captol i or lower case L that is the only thing i can see that is different.
 
My All-In-One was build a year ago and I am in need of additional disk.
I have a RAIDZ2 vdev with six Hitachi GST Deskstar 5K3000 HDS5C3020ALA632 (0F12117) 2TB SATA 6.0Gb drives attached to a M1015. I have ordered another M1015 for the next set of drives.

ESXi 4.1.1
OI 151a5 - nappit 0.8
SM X9SCM-F-O
E3-1230
32GB RAM
Norco 4220

Q1: I am not sure if I am going to use six or eight 2TB or 3TB drives in the new vdev. Does the new raidz2 vdev have have to be the the same size (I don't believe so, and this is not a mirror, but an expansion of the zpool)?

Q2: The currents disks in my vdev are 512 sectors. Is it still recommended to use drives that have the 512 sectors?

Q3: Are there certain formatting commands that need to be run on 4K drives in ZFS
 
My All-In-One was build a year ago and I am in need of additional disk.

Q1: I am not sure if I am going to use six or eight 2TB or 3TB drives in the new vdev. Does the new raidz2 vdev have have to be the the same size (I don't believe so, and this is not a mirror, but an expansion of the zpool)?

Q2: The currents disks in my vdev are 512 sectors. Is it still recommended to use drives that have the 512 sectors?

Q3: Are there certain formatting commands that need to be run on 4K drives in ZFS

q1
You can add any vdev type but its efficient to use vdevs that are similar in size and number of disks.

q2
Ashift is a vdev property, so you can mix. If possible, use 512 sector disks or disks that report 4096 physical sectors. If you buy 2 TB disks, you may buy the same (add a hotspare). 3 and 4 TB disks are all 4k.

q3
when the disks (newer ones does) report the correct physical sectorsize, ZFS will use ashift=12 automatically. With newer OI versions you can try to force ashift via sd.conf (suggested) or use a modified zpool (highly experimental) for older 4k disks that report a wrong physical sector size.

http://wiki.openindiana.org/pages/viewpage.action?pageId=4883847
 
I am trying to install OI on a new VM, I've been able to install on a different server, but the mobo went bad, had to move it yada yada....but now it cant seem to find the hard drive to install it on. Just keeps searching for my SSD, and I know its good. I'm installing it as a Solaris 11 64bit VM, following the all-in-one guide here.

It seems to be an installation bug, anyone else have this problem?

http://docs.oracle.com/cd/E19963-01/html/821-1479/gjsfl.html#gjtlt

I am trying to figure out how to fix, but I can't seem to find good information, and my knowledge is quite lacking in this world.
 
Hi Gea,
This night I received the following alert mail:

Alert/ Error on maneframe from :

-disk errors: none

------------------------------
zpool list
NAME SIZE ALLOC FREE EXPANDSZ CAP DEDUP HEALTH ALTROOT
rpool1 149G 11.3G 138G - 7% 1.00x ONLINE -
tank 4.53T 2.72T 1.81T - 60% 1.00x ONLINE -

The disks and pools seem to be o.k., I manually scrubbed the pool with no alerts...

Thanks for your input!

-Frozen
 
is anyone using Link Aggregation on napp-it?
I would like to use it with this motherboard if possible
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182253

I tried this when I started my project with Sol 11 11/11 to no avail. It was not so easy to make it run under Solaris, as NWAM kept interfering with manually configuring the LACP. I got it running with the help of a script in the end, but it was really not a neat solution. I think it depends on what you want to get from link aggregation. If you want a failover, fine, that's a goal worth chasing. If you go for performance, I'm almost sure you will not get much out of it compared to the hassle to make it run. I turned down the idea at some point, and configured two independent NIC's, then distributed my VM-Box to use one IP, and my PC's to use the other IP for CIFS traffic. That serves my needs perfectly.

By the way, I don't think napp-it cares a lot about aggregated links, it will probably just treat it as one NIC, I guess? It's more of a Solaris thing (or OI, I don't know what you want to build).

Hth,
Cap'
 
is anyone using Link Aggregation on napp-it?
I would like to use it with this motherboard if possible
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182253

Yes, Link Aggregation works perfectly on Solaris 10, 11express and 11. I am running it on my Solaris 11 napp-it based server using 10Gbe links and it works perfectly. As a side note, VLANs over aggregated links work perfectly too.

As Captainquark notes, it is something of a PITA to get set up. The real key is to turn off Network Auto Magic (NWAM). Until you have a stable config, NWAM will just quietly undo every manual change you make to the network configuration - which isn't real helpful unless you have a plain vanilla setup (duh!) or can make your changes in a single command line (which for LAG you cannot do). And the other PITA with doing this is that you need to turn off NWAM from the console - not from an SSH session - because as soon as you turn it off you lose all of the network connectivity. Once you get your LAG set up properly you can turn NWAM back on and it should leave things alone. But you may find - as I did - that you learn enough in the process to not need/want NWAM at all. And it will remember your config across reboots just fine.

There are dozens of tutorials on line about setting this up. Just google "setup LACP solaris 11". Much of what you will find is actually written for Solaris 10, but the core commands have not changed. Just remember to disable NWAM first.

To disable NWAM:
svcadm disable network/physical:nwam
svcadm enable network/physical:default

Additional note here for Solaris 11 11/11: http://blog.jim80.net/2012/01/14/solaris-11-1111-disable-nwam-revisited/
 
Last edited:
Can anyone here state the requirements and explain how to implement napp-it Open Indiana with MPIO using dual gigabit nic adapters so that the throughput across the network to the initiator is 220MBps +.

Both if the initiator is a Windows Server 2008 R2 and if the initiator is ESXI 5.1 (the setup is different)

I've been looking around on the web and the answer is as clear as mud.
 
q1
You can add any vdev type but its efficient to use vdevs that are similar in size and number of disks.

q2
Ashift is a vdev property, so you can mix. If possible, use 512 sector disks or disks that report 4096 physical sectors. If you buy 2 TB disks, you may buy the same (add a hotspare). 3 and 4 TB disks are all 4k.

q3
when the disks (newer ones does) report the correct physical sectorsize, ZFS will use ashift=12 automatically. With newer OI versions you can try to force ashift via sd.conf (suggested) or use a modified zpool (highly experimental) for older 4k disks that report a wrong physical sector size.

http://wiki.openindiana.org/pages/viewpage.action?pageId=4883847

Gea,
Thank you for your response. I have two additional questions.
1. There are not a lot of options for 2TB 512b drives. I am looking at the Hitachi 7K3000 (0F12115) http://www.amazon.com/HGST-Deskstar...e=UTF8&qid=1351268830&sr=1-1&keywords=0F12115
Do you see any concerns?
2. I will probably get a couple of spares, but in the case that I had no spares and had a drive failure (512b sector) and could only replace it with a 4k sector drive, what actions would I have to take to the vdev or pool, either prior or after the physical replacement.
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
Hey!

I just installed Solaris 11.1 which has native zfs version 34. With napp-it on top on of it, my zpool version is 28. What would happen if upgrade I zpool to v.34? Will I loose any of napp-it functionality?
 
Gea,
Thank you for your response. I have two additional questions.
1. There are not a lot of options for 2TB 512b drives. I am looking at the Hitachi 7K3000 (0F12115) http://www.amazon.com/HGST-Deskstar...e=UTF8&qid=1351268830&sr=1-1&keywords=0F12115
Do you see any concerns?
2. I will probably get a couple of spares, but in the case that I had no spares and had a drive failure (512b sector) and could only replace it with a 4k sector drive, what actions would I have to take to the vdev or pool, either prior or after the physical replacement.

1.
7k3000 are fine

2.
You can replace a 512 drive with a 4k one (at a slightly reduced performance) mostly without extra action. But there are reports that a replacement is sometimes refused.
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
Hey!

I just installed Solaris 11.1 which has native zfs version 34. With napp-it on top on of it, my zpool version is 28. What would happen if upgrade I zpool to v.34? Will I loose any of napp-it functionality?

No, you will only loose compatibility with the free Illumos based alternatives.
 
Having problem with Solaris 11.1 after installing nappit.
When I try to change root password as said in installation guide I get following message:
Code:
# passwd root

Please try again later.

Please try again later. 
Permission denied.
Neither I can set passwords for local users via web interface.
 
Having problem with Solaris 11.1 after installing nappit.
When I try to change root password as said in installation guide I get following message:
Code:
# passwd root

Please try again later.

Please try again later. 
Permission denied.
Neither I can set passwords for local users via web interface.

Oh Oracle - why do you change everything -
####################################
(they moved pam settings to a different location)

to fix it:
Code:
edit /etc/pam.conf and delete the line
other   password required   pam_smb_passwd.so.1 nowarn

edit /etc/pam.d/other and add
password required   pam_smb_passwd.so.1 nowarn

I have updated the installer to care about
 
Dear Gea,
shall I keep experimenting with Solaris 11.1 or roll back to 11/11? Because I have a feeling thats not the last thing they have changed...
Also what is the most adapted, stable and problem free distribution for napp-it?
 
Oh Oracle - why do you change everything -
####################################
(they moved pam settings to a different location)

to fix it:
Code:
edit /etc/pam.conf and delete the line
other   password required   pam_smb_passwd.so.1 nowarn

edit /etc/pam.d/other and add
password required   pam_smb_passwd.so.1 nowarn

I have updated the installer to care about
Are you saying napp-it messes with system files?

It's not "Oh Oracle - why do you change everything" it's "Oh Gea - why do you mess with the system without checks".
 
People participating in this thread and this topic in general need to understand that Solaris is now different from illumos. For the purposes of this thread you should consider them incompatible. And since napp is well tested on only illumos, solaris is dead to most people here
 
Are you saying napp-it messes with system files?

It's not "Oh Oracle - why do you change everything" it's "Oh Gea - why do you mess with the system without checks".


Evey time when you try a new major software release that is out less than a week, you must be aware that things have changed. That is normal and therefor you must do some tests prior use. (I basically installed Solaris 11.1 yesterday the first time to check this behaviour - its not my daily use platform that is OpenIndiana)

My "Oh Oracle" is due to the very minor modifications that breaks essential howtos. Up to Solaris 11 Express, the basic system administration was nearly identical to OpenSolaris and the Illumos family. With Solaris 11.11 they for example renamed a basic command for network settings from a former ipadm xxx-if to ipadm xxx-ip with the effect that all former howtos did not work any longer.

This is the same with the current modification (move a other .. entry in pam.conf to a separate other file). This pam setting is the only needed modification for a working CIFS server (the most used service on Solaris). Any former howto setup CIFS is now worthless The same with the current Oracle docs when you use older Solaris (Express) 11 or Illumos based ones (this may be intended).

And yes, napp-it modifies system files during setup, basically /etc/sudoers to allow napp-it system administration and /etc/pam.conf (/etc/pam.d/other on Solaris 11.1 now) to enable the CIFS server.
 
Last edited:
Alrighty then. It seems OI is nobrainer right now for using in sake of nappit. But I kinda concerned of its future development because of latest events (project leader resign). Any downsides of using Illumos instead?
 
Alrighty then. It seems OI is nobrainer right now for using in sake of nappit. But I kinda concerned of its future development because of latest events (project leader resign). Any downsides of using Illumos instead?

Illumos is not a OS that you can use - It is the project to maintain the kernel and basic tools to build a OS on top like Eon, Illumian, OI, OmiOS or SmartOS does.

So the latter are your options (napp-it is currently running on OI, Illumian (partly) and Solaris 11)
 
I tried this when I started my project with Sol 11 11/11 to no avail. It was not so easy to make it run under Solaris, as NWAM kept interfering with manually configuring the LACP. I got it running with the help of a script in the end, but it was really not a neat solution. I think it depends on what you want to get from link aggregation. If you want a failover, fine, that's a goal worth chasing. If you go for performance, I'm almost sure you will not get much out of it compared to the hassle to make it run. I turned down the idea at some point, and configured two independent NIC's, then distributed my VM-Box to use one IP, and my PC's to use the other IP for CIFS traffic. That serves my needs perfectly.

By the way, I don't think napp-it cares a lot about aggregated links, it will probably just treat it as one NIC, I guess? It's more of a Solaris thing (or OI, I don't know what you want to build).

Hth,
Cap'
Aggregated links if you have LACP (and Giant Frames) should increase the throughput for large files.
I had to read up on Solaris's manual to get aggr links to keep their settings after reboot with 11.1
http://docs.oracle.com/cd/E23824_01/html/E24456/gliyc.html#OSTELgllcu
http://docs.oracle.com/cd/E23824_01/html/821-1458/fpjvl.html
Instead of disabling nwam I set it to the DefaultFixed profile
Code:
# netadm enable -p ncp DefaultFixed
Had to use svccfg commands to get things going.
I tested it in a vm before I tried it on my SAN.
 
Last edited:
I've been running Napp-It in production for a few weeks now to host our PHD Virtual backups. So far I cannot complain about the quick fix I had to implement to keep us from more expensive disks on our MD3000i. The only issue we have now is that our backups take forever due to the increased space, so I am wondering if anyone has any info on integrating our Dell TL-2000 tape array to the Napp-It server to handle direct backups instead of going through BackupExec and the VMware adapters. What is everyone else doing for backups of their Napp-It servers?
 
Me backups/archive at Tanndberg T24 +Win2003 NT Backup via CIFS/SMB. Little bit a twisted situation :)


EDIT:
At Solaris 11.1 ,napp-it ->System is able to show the rpool's status,hip hip hoooray!!!
That's the first "issue" which I found till now,testing combination of Solaris11.1+napp-it.
More of this ,please :)

The setup was/is,in order :
VMware Workstation,Solaris 11/11 text install,pkg install solaris-desktop,
made the rpool mirrored and bootable,install napp-it ,afp and setup raidz pool v28 with some basic CIFS config.
Update the OS via Gnome's "Update Manager",update process took two passes - 317MB and 881MB ,separated by one reboot.
 
Last edited:
This probably is not a napp-it problem but just in case it is my question is I had a pool named thor that I just cleared off and destroyed with no problem. I am reconfiguring from a raidz2 setup to a raidz setup. When I try to create pool with the same name "thor" it says "pool /thor already exist!". I checked zpool list and it is not listed nor is it listed in napp-it's pool list.

Is it as simple as I need to delete the folder under solaris so it does not think the pool exists or is there another process I need to follow to start a new pool with the same name as the old one?


Edit: Upon further inspection I found under import:

Code:
exported pools

   pool: thor
     id: 14547134680173632568
  state: FAULTED
 status: The pool metadata is corrupted.
 action: The pool cannot be imported due to damaged devices or data.
   see: http://illumos.org/msg/ZFS-8000-72
 config:

	thor        FAULTED  corrupted data
	  c5t3d0p0  ONLINE
	  c5t2d0    ONLINE

destroyed pools:

   pool: thor
     id: 9764473289643050985
  state: ONLINE (DESTROYED)
 action: The pool can be imported using its name or numeric identifier.
 config:

	thor         ONLINE
	  raidz2-0   ONLINE
	    c6t14d1  ONLINE
	    c6t16d1  ONLINE
	    c6t15d1  ONLINE
	    c6t9d1   ONLINE
	    c6t10d1  ONLINE
	    c6t11d1  ONLINE
	  raidz2-1   ONLINE
	    c5t1d0   ONLINE
	    c5t0d0   ONLINE
	    c5t7d0   ONLINE
	    c5t6d0   ONLINE
	    c5t5d0   ONLINE
	    c5t4d0   ONLINE

Will it not let me recreate because of the faulted old pool id? How do I make these go away so I can create a new pool named thor?
 
Last edited:
Back
Top