OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

arryo

n00b
Joined
May 23, 2012
Messages
55
If this happens without any other modification,
i would expect a problem with your OmniOS setup

I would try
- boot into a former boot environment or
- update /reinstall newest 151014 long term stable

http://omnios.omniti.com/wiki.php/ReleaseNotes/r151014
http://omnios.omniti.com/wiki.php/Upgrade_to_r151014


I did a reinstall of omnios 151014 and napp-it and everything is fine now. It's weird. Now my HDDs not spin down anymore even though I already set in power management for them to spin down after 900s
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,065
There is no modification in napp-it 0.95 and no known problem with OmniOS.

Have you disabled monitoring and background acceleration (upper right menu)?
Optionally disable fmd (fault management service) and alerts.
 

arryo

n00b
Joined
May 23, 2012
Messages
55
I don't think i disable monitoring, in fact I wouldn't know how to. How do I disable fmd and alerts?

One thing is, when I unplug my 2 M1015 cards, letting only 1 card running, the disks are spinning down, but when I use all 3, none of the disks is spinning down.

update 5/18:

The disks are spinning down fine with 1 or 2 cards added in but not all 3 cards. To discard the faulty with the card/disks attached to each card i tested the combination with any two cards and the disks are spinning fine with any 2 cards in place. So I think there's something going on that prevents the system work properly with 3 cards.
 
Last edited:

CopyRunStart

Limp Gawd
Joined
Apr 3, 2014
Messages
155
Hey Gea, still having replication issues.

I deleted the pool on the target server and deleted all snaps related to replication on both source and target.

I then re-did the initial replication and it worked fine. As soon as I tried to do a 2nd replication, I had the same error as before:

info: 651: incremental zfs receive error after 5 s cannot receive: most recent snapshot of backup/priv does not match incremental source
src snaps: private/priv@1431619496_repli_zfs_frontend-02_nr_1 -> private/priv@1431619496_repli_zfs_frontend-02_nr_2
dest: backup/priv with last snap private/priv@1431619496_repli_zfs_frontend-02_nr_1
Maybe you have autosnaps with delzero snaps=yes -you should not activate this option on replicated filesystems-
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,065
I don't think i disable monitoring, in fact I wouldn't know how to. How do I disable fmd and alerts?

Fmd (fault managemen service) is a Solaris service to check health.
You can disable with svcadm disable fmd

Alert is a napp-it autojob that sends alerts on pool problems.
You must enable manually.
 

tobiasl

Weaksauce
Joined
Nov 19, 2007
Messages
100
Hi sorry to ask so stupid, but like to be sure I don't do any thing wrong.
I got a zfs folder shere over smb I like to get all user read and white access to how do I do that
zfs set sharesmb=on tank/Ipad and what more ? sorry from the noob :)
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,065
Hi sorry to ask so stupid, but like to be sure I don't do any thing wrong.
I got a zfs folder shere over smb I like to get all user read and white access to how do I do that zfs set sharesmb=on tank/Ipad and what more ? sorry from the noob :)

Per default, only the owner/creator (root) has access, so you must set permissions
-if you use napp-it, everyone@=modify (what you want) is the default when you create and SMB share a filesystem, another option is the napp-it acl-extension
- you can also use Windows Pro to set permissions (connect as root)
- last option is /usr/bin/chmod (set ACL at console)
 

Captainquark

Weaksauce
Joined
Dec 14, 2011
Messages
103
Hi Gea,
I have two boxes, one OmniOS with Napp-it 0.9f3 and one VMWare ESXi 5.5 (Build 2068190). I upgraded ESXi to 5.5 a few month ago, and my NFS share always worked great. Now I had a faulty disk lately and decided to shut down the NAS to replace it (I can't see the WWN from the outside of the case, so that's the fastest way to do it properly, I guess). So I put the ESXi box in maintenance mode, shut down the NAS, replaced the disk, powered the NAS back on and replaced the faulty with the new one in Napp-It. All of this worked great. Then I took the ESXi out of maintenance mode, and it couldn't see the NFS share anymore. Tried rebooting to no avail. Removed the store and tried to add it again, and I get the error message:
...The NFS Server does not support MOUNT version 3 over TCP.
I found it very odd to happen, as the ESXi and the NAS were running fine for at least 4 months in exactly this configuration!
Found then another article at servethehome.com where a guy asks you the same, and you advise to update ESXi to "update 2", which solved his problem. So I tried this as well, but it did not help. I am running now ESXi 5.5 Build 2403361, which should be Update 2.
So I had another look at Napp-it, and noticed that even though in the ZFS Filesystem, my share is NFS=On (Permission is every@=mod), the NFS Server Service = offline. I tried setting it online several times, and I get no error message there, but it remains offline permanently. Any idea what the problem could be, and how to solve this?
Thanks a lot for your support.
Kind regards,
Cap'
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,065
There are two problems with ESXi 5.5
5.5: e1000 not stable
5.5 U1 NFS bug

5.5 U2 very stable

If NFS service stays offline without any reason while it worked before,
I would expect that something happens with your setup

I would
- try a former BE or
- update to OmniOS 151014 LTS if your OmniOS is older
 

Captainquark

Weaksauce
Joined
Dec 14, 2011
Messages
103
Thanks _Gea for your quick response.
I had a quick look at the OmniOS console, and found this:
Failed to connect to local statd (tpcerr=5)
svc.startd[10]: svc:/network/nfs/nlockmgr:default: Method "/lib/svc/method/nlockmgr" failed with exit status 1.
Failed to connect to local statd (tpcerr=5)
svc.startd[10]: svc:/network/nfs/nlockmgr:default: Method "/lib/svc/method/nlockmgr" failed with exit status 1.
Failed to connect to local statd (tpcerr=5)
svc.startd[10]: svc:/network/nfs/nlockmgr:default: failed: transitioned to maintenance
mDNSResponder: ERROR: getOptRdata - unknown opt 65002
I have to say I'm a complete noob when it comes to OmniOS... so do you think an upgrade would help solving the issue? If so, how do I do an upgrade of OmniOS? If I am not mistaken, I run omnios-10b9c79 today.
Thanks,
Cap'
 

Captainquark

Weaksauce
Joined
Dec 14, 2011
Messages
103
Thanks, _Gea, updating to 151014 did solve the Problem... weird, no idea what caused it, though.

Anyway, once more, saved me, so... appreciated!
Kind regards,
Cap'
 

ldoodle

Limp Gawd
Joined
Jun 29, 2011
Messages
172
Hey,

I need to change a scheduled job (scrub). I can't for the life of me remember which file I need to edit!

I'm using:

OpenIndiana (powered by illumos) SunOS 5.11 oi_151a September 2011

Thanks
 

Aesma

[H]ard|Gawd
Joined
Mar 24, 2010
Messages
1,854
My two cards are now in P19 firmware, seem to be working fine.

I keep my OS up to date since I've installed it two years ago (OpenIndiana), although recently I'm back to one of the oldest boot environments because none of the others manage to boot. OI is offering me 895 updates, and I don't know which one is causing this !
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,065
I would do a re-install of the newest OI or Hipster dev edition

- and would prefer a switch to a stable OmniOS 151014 LTS as OI/ Hipster development is quite slow and may never reach a stable state with continous bug and security fixes.
 

Aesma

[H]ard|Gawd
Joined
Mar 24, 2010
Messages
1,854
The thing is, at least until I go Infiniband or 10GbE, I do several file operations directly on the server so as to not be limited by 1GbE, so I need a desktop for now. And I've installed several programs, some windows ones through wine, it took me a while to figure out so I'm wary of starting from scratch.
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,065
You can copy/move files locally with a filemanager like Midnight Commander.
You can try with your setup as well, start it with mc at console or better remote via Putty (sudo mc if you are not root).

If you need Windows and other VMs, All-In-One based on ESXi would be a perfect solution.
 

ldoodle

Limp Gawd
Joined
Jun 29, 2011
Messages
172
cron ?
napp-it, which version?

btw
You should think about an update

cron. I use the vanilla install.

Yeah, I know I need to do an update - just nervous about doing it and it killing my data!

PS: How comes all the updates for OI are 'prestable'. Where's the 'stable' ones?
 
Last edited:

ldoodle

Limp Gawd
Joined
Jun 29, 2011
Messages
172
Thanks. I've seen that page, but it doesn't say what the default file is.

I didn't create my own one, just modified the default built-in (blank) one!!
 

cbutters

Gawd
Joined
Dec 30, 2005
Messages
514
Quick theoretical question..
Lets say that I'm running an all-in-one esxi box with storage passthrough on an LSI card, and the omnios vm on the boot drive (esxi datastore) just disappears / goes corrupt / the esxi datastore drive goes bad.

Should the recovery process be A or B:?

A) Assumes a daily Omni-OS VM Backup is saved.
1: Install a new datastore drive for ESXi
2: Load the backup of that omni-os VM onto the datastore.
3: Boot the VM and all my pools will show up.

B) Assuming no backups of the omnios vm.
1: Install a new datastore drive for ESXi
2: Roll a new Omni-OS VM (Fresh, not a backup)
3: Boot the VM and run a zpool import and the pools will show up.

Which is better? Scenario A or B?
Will there be a data congruency issue with A? like the data wont line up quite right?
Thanks
 
Last edited:

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,065
You can do both.
I use B as a pure storage OmniOS installation is not worth any backup efforts.

I do backups of
- napp-it settings like groups or jobs
- user
- idmappings

from 0.9f6 on there is an autojob for this otherwise backup at least /var/web-gui/_log
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,065
Hi
Having trouble getting the TLS email function working. I have followed the instructions here - http://napp-it.org/downloads/tls_en.html and even tried the bug fix at the bottom of that web page.
Napp-it ver. 0.9f5 Apr.22.2015 OmniOS ver. r151014

Getting error-
Software error:
Couldn't start TLS: SSL Version SSLv3 not supported
at /var/web-gui/data/napp-it/zfsos/15_jobs and data services/04_TLS Email/09_TLS-test/action.pl line 72.

Any ideas?

fixed in 0.9f5 and 0.9f6 from May 28
https://rt.cpan.org/Public/Bug/Display.html?id=77401
 

cbutters

Gawd
Joined
Dec 30, 2005
Messages
514
You can do both.
I use B as a pure storage OmniOS installation is not worth any backup efforts.

I do backups of
- napp-it settings like groups or jobs
- user
- idmappings

from 0.9f6 on there is an autojob for this otherwise backup at least /var/web-gui/_log

Thanks for the response. Very helpful. I was debating whether buying another ssd and mirroring the OS datastore Drive would be worth it. Seems like it would only be worth it if you had high uptime demand. In my home server use case, if it dies, I just get a new drive, reload a fresh omnios-vm, reimport the pool and my settings.

Thanks.
 

cbutters

Gawd
Joined
Dec 30, 2005
Messages
514
Here is a very unscientific, but fun comparison of the affects of Memory on ZFS and also the efficiency of file copy methods that I'm seeing as I transfer files to my new ZFS build.

This is the same data set, and a visualization of the ability of the transfer to saturate the gigabit network.

Source: RAID6 8x4TB Array (should be fully capable of saturating the array (It can do 900MB/s+)

Destination: Network ZFS OmniOS/napp-it 8x6TB RAIDZ2 on LSI Passthrough

zfs3.png
 

ldoodle

Limp Gawd
Joined
Jun 29, 2011
Messages
172
If you need a stable Illumos distribution with regular fixes, use OmniOS

Can I migrate to OmniOS without having to move my data off and back on?

Is this what pool exporting/importing is for? Never done it before so a bit nervous about that too!
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,065
Can I migrate to OmniOS without having to move my data off and back on?

Is this what pool exporting/importing is for? Never done it before so a bit nervous about that too!

NexentaStor, OpenIndiana, OmniOS and SmartOS are all Illumos distributions.
You can always move pools between them (at least from older to newer OS releases
as the target OS must support same pool version or feature flags)

From any OpenIndiana to current OmniOS 151014: zero problems
Export the pool in Openindiana (not a must) and import in OmniOS

OmniOS setup, see http://napp-it.org/downloads/omnios_en.html
 

ldoodle

Limp Gawd
Joined
Jun 29, 2011
Messages
172
From any OpenIndiana to current OmniOS 151014: zero problems
Export the pool in Openindiana (not a must) and import in OmniOS

Thanks. What I've never understood is how export actually works!

1) Export the pool
2) Kill the OI install (wipe disks)
3) Install OmniOS
4) Import the pool

If I kill the OI install, how can the pool be re-imported? Does the export 'data' get stored on the pool itself?
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,710
There is no 'export data' per-se. If you say 'import tank', it will scan any disks present, and if it finds a valid pool with that name, it will import it.
 

shanester

Weaksauce
Joined
Mar 1, 2011
Messages
70
Yesterday, after a reboot to swap out a fan, it appears that disaster has struck and need some advice on what my options and/or next steps should be.

I am running a all-in-one with OmniOS r151012

There are several issues.
1. There is almost no disk space left in rpool (I removed the crash dump) and I can't determine what is taking up the space
2. zfs pool now shows that the pool is unavailable with no space left (but in actuality there does appear to be space (see below).

SzTgUYh.png


G00OV4H.png


JpbMQhC.png
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,065
I would first check if you have some BE that you can delete,
then check /var/adm for large logfiles due errors (move/read/delete them)

If you have another server, shut down, plug in the disks there to rule out hardware problems

or
shut down, check for a PSU or cabling problems (check connectors).

or
wait until the resilvering is ready but I would power off and check cablings and connectors.

The good: If unavail disks come back, your pool will come back.
 

shanester

Weaksauce
Joined
Mar 1, 2011
Messages
70
I would first check if you have some BE that you can delete,
then check /var/adm for large logfiles due errors (move/read/delete them)

If you have another server, shut down, plug in the disks there to rule out hardware problems

or
shut down, check for a PSU or cabling problems (check connectors).

or
wait until the resilvering is ready but I would power off and check cablings and connectors.

The good: If unavail disks come back, your pool will come back.

Gea,
I thought I would try a fresh install of omni and then attempt to import the pool. Here is a screen shot of the options. I am not sure what I should do based upon the overview.
Based on this info what do you suggest? I am going to check the cabling again.

OaAabF4.png
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,065
Raid Z2-0 vdev has 3 unavailable disks.
This is more than your redundancy level (max 2 failed disks)
so you cannot import unless you can fix this.

As half of your disks are unavail, check cabling or HBA
 

g0dM@n

Supreme [H]ardness
Joined
Nov 18, 2005
Messages
5,365
shanester, not that I could really help, but did you figure this out?
I know if this happened to me, I wouldn't leave the house until I fixed it.
 
Joined
Dec 30, 2010
Messages
43
Gea,
Pushover isn't available for Windows Phone users. I have no idea how much work it is, but can you add support for Pushalot as well? It is something like pushover. It has apps for Windows 8 and Windows Phone. And it is totally free.

I'm making plans to build another nappit box. thinking of using the HP Proliant MicroServer Gen8 G1610T. Does anyone have experience using omnios & nappit on those machines?
I built a few years ago 2x HP Proliant Microserver N40L and never had any problems with them so far. Nice machines.
 
Top