OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

_Gea Yes, I have added it and it works. Do you know about any good tutorial on doing iSCSI back presentation from ZFS napp-it to ESXi ? I am pretty green on both ZFS and iSCSI. I mainly used DAS. I do have iSCSI enabled in ZFS base settings on my SSDs pool. I would highly appreciate an advise that would cover some basic steps. As this is all-in-one-device setup do I need a vSwitch with Jumbro frames? This are the types of things I still do not know.

Its all virtual, there is plenty of documentation, but for a physical setup. Unfortuantely.
 
Last edited:
_Gea Yes, I have added it and it works. Do you know about any good tutorial on doing iSCSI back presentation from ZFS napp-it to ESXi ? I am pretty green on both ZFS and iSCSI. I mainly used DAS. I do have iSCSI enabled in ZFS base settings on my SSDs pool. I would highly appreciate an advise that would cover some basic steps. As this is all-in-one-device setup do I need a vSwitch with Jumbro frames? This are the types of things I still do not know.

Its all virtual, there is plenty of documentation, but for a physical setup. Unfortuantely.
So far I have:
MicroServer
Xeon E3-1230, 16GB ram, SATA SSD 120GB in optical bay, SD card, H200 ITmode w/ 2xSSD230GB,2xHDD2TB attached
ESXi 6.5 is up and running, boots from SD
napp it is installed, H200 is assigned to it, SSD and HDD mirrors are created, iSCSI is enabled, targets and views are created
thats the moment where I am stuck
 
So far I have:
MicroServer
Xeon E3-1230, 16GB ram, SATA SSD 120GB in optical bay, SD card, H200 ITmode w/ 2xSSD230GB,2xHDD2TB attached
ESXi 6.5 is up and running, boots from SD
napp it is installed, H200 is assigned to it, SSD and HDD mirrors are created, iSCSI is enabled, targets and views are created
thats the moment where I am stuck
Warning - it's been a while since I've done iSCSI, and longer for Comstar.

Go to ESXi, Configure, Storage Adapters. You need to (add &) configure the iSCSI Software Adapter. It also needs to be bound to an ESXi network, and using a separate net can't hurt.
 
Warning - it's been a while since I've done iSCSI, and longer for Comstar.

Go to ESXi, Configure, Storage Adapters. You need to (add &) configure the iSCSI Software Adapter. It also needs to be bound to an ESXi network, and using a separate net can't hurt.

I still cant see any targets. I have added a Virtual network adapter to napp-IT comstar enabled machine, added a network IP address, it is seen as 10G Ethernet, but I think I am missing something that ESXi could see storage presented by a VM hosted on that ESXi.
 
Without a very special reason, I always prefer NFS.
Setup is no more than enable NFS in OmniOS and add NFS storage in ESXi.
It is uncritical against delayed availabilty and you have parallel access via SMB and Windows Previous Versions for copy/clone/move/backup

With same setting (sync vs writeback cache) NFS is similar regarding performance
 
Propably you need to add a persistent default gateway
https://docs.oracle.com/cd/E53394_01/html/E54745/gmyag.html

Thank you Gea, that was what the issue was. I do have a follow up question on this.

upload_2017-2-4_12-54-30.png


When I look at the netstat table, I can see the default gateway listed, but the aggregation (aggr0) is still listed incorrectly. The static IP is listed as the gateway and the destination is incorrect for both the gateway and static ip. When I run, "route get default" it properly list the aggregation correctly as the default route and includes the correct gateway. I have a feeling this is mixed up, but it is still technically working. How can I adjust the aggregation to have the correct information? I looked through some of the commands and couldn't find the correct angle to look at this. Thanks.
 
Have you tried to flush/delete all entries an re-create the default route only.

# route flush
# route -p add default ip-address
 
Gea,

I have a few questions for you mainly around subscriptions and licensing. First and foremost I want to thank you for napp-it and everything you've done for the community. I recently read your performance paper surrounding the new versions of OmniOS and Solaris and SMB2.1. It motivated me to move to the latest bloody version of OmniOS where I have seen improved throughput which is encouraging. To further optimize my setup I wanted to try the Appliance Tuning feature but even with the Pro trial it does not seem to be included. Am I doing something wrong or is that accurate?

With that said I believe I can afford to to purchase a napp-it Pro single extension license. With this licensing tier is Appliance Tuning included and if so does it count as the one included extension?

As always thanks!
 
Anybody tested OmniOS and Napp-it under Hyper-V? Might think of moving to Hyper-V soon due to licensing, so was wondering if this is feasible with OmniOS.
 
Gea,

I have a few questions for you mainly around subscriptions and licensing. First and foremost I want to thank you for napp-it and everything you've done for the community. I recently read your performance paper surrounding the new versions of OmniOS and Solaris and SMB2.1. It motivated me to move to the latest bloody version of OmniOS where I have seen improved throughput which is encouraging. To further optimize my setup I wanted to try the Appliance Tuning feature but even with the Pro trial it does not seem to be included. Am I doing something wrong or is that accurate?

With that said I believe I can afford to to purchase a napp-it Pro single extension license. With this licensing tier is Appliance Tuning included and if so does it count as the one included extension?

As always thanks!

All appliance menus (Disk Map. Security, Tuning) are part of the full/complete license.
The other rerstriction is that they do not work on a Free software edition.
Even with a key you must update to Pro.

Your options
Set tuning options like tcp buffers, NFS servers or Jumbo manually an console or via WinSCP:
You can set similar values like http://napp-it.org/doc/downloads/napp-it.pdf page 42

With a single extension, you can update to napp-it Pro.
The tuning menu should work then with an evalkey that you can request online at napp-it.org
 
Anybody tested OmniOS and Napp-it under Hyper-V? Might think of moving to Hyper-V soon due to licensing, so was wondering if this is feasible with OmniOS.

I have not tried but this should work.
With Hyper-V 2012 performance may be restricted by the quality of the vnic drivers. I would not expect vmxnet3s performance.
As you cannot pass-through nics, HBA and disks you must use disk mapping (comparable to ESXi RDM).
You must check performance and if you can read Smart values. If any write cache in Hyper-V is involved it may compromise sync write behaviour of ZFS

Hyper-V 2016 is announced to have pass-through but I have no further info.
 
Last edited:
Anybody tested OmniOS and Napp-it under Hyper-V? Might think of moving to Hyper-V soon due to licensing, so was wondering if this is feasible with OmniOS.

I did a bit of testing. Omnios did not work on a generation 2 vm under Hyper-V because of the UEFI firmware in generation 2 vm. Generation 1 vm has a maximum of 4 virtual disks. Passthrough of HBA is not supported so you end up with a maximum of 3 mapped disks and 1 bootdisk.
This was not enough for my typical storage configuration so I gave up on testing.
 
_Gea & toelie

Ok, thanks for confirming. Looks like its a no go, as my entire zfs environment is centered around passing-through 2 m1015s, which then handles the disks. Maybe once 2016 matures, then I can look at it again...
 
All appliance menus (Disk Map. Security, Tuning) are part of the full/complete license.
The other rerstriction is that they do not work on a Free software edition.
Even with a key you must update to Pro.

Your options
Set tuning options like tcp buffers, NFS servers or Jumbo manually an console or via WinSCP:
You can set similar values like http://napp-it.org/doc/downloads/napp-it.pdf page 42

With a single extension, you can update to napp-it Pro.
The tuning menu should work then with an evalkey that you can request online at napp-it.org

Thanks for the insight. After inputting the eval key I went to the update menu and I did not see any options for downloading the pro versions of Napp-It. The main menu does show that I'm in a day trial so it looks as if the key was installed successfully. I'll give it another try tonight. Thanks again!
 
Updates to a napp-Pro or Dev edition is not possible from the Free edition with an evalkey.
 
1. lx zones are beta in 151020 and must be installed manually via
pkg install brand/lx

2.
This should be fixed in 17.01dev and due the removal of the Javascript driven "MBmenu" in 17.01
 
1. lx zones are beta in 151020 and must be installed manually via

I'm not sure I understand.

If I go to Services >> Zones, I see:

zones service: online
zones monitoring: online

But on the About page, I see:

zones service : disabled

Is lx zones affecting one place but not the other?
 
There was a bug in the About > service overview (fixed in newest dev).
The Service menu where you can switch state is correct
 
Hey Gea.

I'm running Napp-it on a few servers, some Pro, some free. On the 17.01 Free server, I only have the option to use Pushover. On the 17.01 Pro server, I only have the option for Pushalot. I need Pushover on all the servers. I have tried multiple browsers but on the 17.01 Pro servers dropdown menu, Pushalot is the only option.
 
Hi

You must first select and set the wanted Push service in About > Settings.
In the job menu you can then create a job with the selected Push service only.
 
I am looking for some zfs assistance. I have been running into some disk errors over the past week.

All-in-One esxi 6.0U2
OMNIOS R20
2 - RAIDZ-2 vdevs w/6 2TB drives in each

This morning the device was unresponsive and when I rebooted the OS went into maintenance mode and the zfs pool is unavailable.

I am not sure how to fix the errors that are displayed in maintenance mode to attempt to resolve the pool errors.
I tried creating a new vm and re-importing the pool, but it failed.

Any suggestions are appreciated.

http://imgur.com/a/AThAV
 
There are two options

Your pool is damaged and the problem occurs when OmniOS tries to auto import the pool.
To check, remove all datapool-disks and reboot.

If OmniOS comes up properly, then one or more disk are the problem.
Re-insert disk by disk and check if it is detected properly (ex via format at console, cancel format after start with ctrl-c).
If all disks come up, import the pool optionally with the -F option (former state).

If OmniOS does not come up properly, try a former BE or reinstall the OS/ reimport a working .ova
and import the pool
 
_Gea, as always thanks for your help. I pulled all the disks, and inserted the first disk, ran format, selected the disk and when the formatting started hit ctrl-c. I'm on the 4th disk and after each format command, the 'searching for disks..' takes longer and longer.

1. Is this normal ?
2. Am I supposed to pull the disk after the format before I insert the next disk?
There are two options

Your pool is damaged and the problem occurs when OmniOS tries to auto import the pool.
To check, remove all datapool-disks and reboot.

If OmniOS comes up properly, then one or more disk are the problem.
Re-insert disk by disk and check if it is detected properly (ex via format at console, cancel format after start with ctrl-c).
If all disks come up, import the pool optionally with the -F option (former state).

If OmniOS does not come up properly, try a former BE or reinstall the OS/ reimport a working .ova
and import the pool
 
You use format only to list all disks (do not format anything).
If format listing hangs on the 4th disk, this disk propably blocks everything.

Reboot without this disk and check if the OS boots with the pool mounted in a degraded state.
 
You use format only to list all disks (do not format anything).
If format listing hangs on the 4th disk, this disk propably blocks everything.

Reboot without this disk and check if the OS boots with the pool mounted in a degraded state.

Thanks. I was able to bring this back online. I definitely have some hardware issue, not sure if it is the M1015s or the backplanes. I the drives look good according to smartmontools. Below is the zpool status and current all-in-one config

I have been pondering the thought of adding additional storage, but the instability of this hardware is making me rethink my options. I think it may be best to get rid of my norco case and go with a used SuperMicro storage chasis or other reputable manufacturer and start over. I put this system together over 5 years ago I am not up to speed with the different hardware architectures.

Recommendations are welcome!!

Requirements:
Used equipment/ebay - keeping expenses down
Minimum of (24) 3.5" sata3/SAS3 hotswap bays
Minimum of (2) 2.5" internal sata drives
Ability to expand into another enclosure
All-in-One OmniOS/Vmware solution
Motherboard should have sufficient PCIs/bus for expand-ability.

I think that I will continue with the smaller drives (2TB or 3TB) in a 10 drive Z2 vdev or 12 drive Z3 vdev, therefore able to continue to utilize my existing vdevs in the new case.


Current All in One Config

HW Config
Supermicro X9SCM-F
Intel E3-1200
32GB RAM
Norco rpc-4220
(2) IBM M1015 fw 20.00.07.00
PCI 4 --> Port 0 -> Backplane 1
Port 1 -> Backplane 2

PCI 5 --> Port 0 -> Backplane 3
Port 1 -> Backplane 4

13 HGST 2TB drives (512k)
4 Drives - Backplane 1
4 Drives - Backplane 2
4 Drives - Backplane 3
1 Drives - Backplane 4

(1) 4 port 82725B Intel GB adapter
PCI 7

SW Config
ESXi 6.0 U2
omnios-r151018-258cc99
Napp-it 17.01 pro

ZFS Pool
(2) RAIDZ2 vdevs with 6 - 2TB drives & 1 hot spare

Primary Use media storage & file share
Most VMs/NFS have been moved off to Thecus NAS due lack of space and vmware APD due to storage instability (I have two additional esxi hosts in a cluster).


zpool status -v
pool: rpool
state: ONLINE
scan: scrub repaired 0 in 0h2m with 0 errors on Sun Feb 12 22:02:38 2017
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
c5t0d0s0 ONLINE 0 0 0

errors: No known data errors

pool: zfs
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Sat Feb 25 13:50:14 2017
14.1T scanned out of 19.5T at 201M/s, 7h48m to go
1.26T resilvered, 72.29% done
config:

NAME STATE READ WRITE CKSUM
zfs DEGRADED 0 0 34
raidz2-0 ONLINE 0 0 0
c8t5000CCA369CAB2A4d0 ONLINE 0 0 0
c8t5000CCA369CD3F3Ad0 ONLINE 0 0 0
replacing-2 ONLINE 0 0 10
c8t5000CCA224C7EEA1d0 ONLINE 0 0 0 (resilvering)
c8t5000CCA36AD05EA2d0 ONLINE 0 0 0 (resilvering)
c8t5000CCA369CD6427d0 ONLINE 0 0 9 (resilvering)
c8t5000CCA369CF0059d0 ONLINE 0 0 0
c8t5000CCA369F7976Ed0 ONLINE 0 0 0
raidz2-1 DEGRADED 0 0 88
c8t5000CCA36AC2A785d0 DEGRADED 0 0 0 too many errors
c8t5000CCA36AC2ADB2d0 DEGRADED 0 0 0 too many errors
c8t5000CCA36AC2BCCAd0 ONLINE 0 0 0 (resilvering)
c8t5000CCA36AC2BCCDd0 ONLINE 0 0 0 (resilvering)
c8t5000CCA36AC2BCDAd0 DEGRADED 0 0 0 too many errors (resilvering)
c8t5000CCA36AC2D771d0 ONLINE 0 0 0 (resilvering)

errors: No known data errors
 
I would expect a problem with cabling, connectors or power.
I would shutdown the system and control all cables and pci adapters if they are inserted properly.

The good thing with ZFS
Even if more disks fail per vdev than are allowed for a raid-level (two in your case), your pool
is back online when enough disks come back. ZFS is in this sense less critical than hardware raid.

To reset the too many error warnings, use zpool clear (napp-it menu pools)
 
I would expect a problem with cabling, connectors or power.
I would shutdown the system and control all cables and pci adapters if they are inserted properly.

The good thing with ZFS
Even if more disks fail per vdev than are allowed for a raid-level (two in your case), your pool
is back online when enough disks come back. ZFS is in this sense less critical than hardware raid.

To reset the too many error warnings, use zpool clear (napp-it menu pools)

The zpool clear worked for a bit ;)
I agree that this is a hardware issue. I did validate/reconnect/replace the SFF-8087 last week. A couple years back I replaced the cables due to similar circumstances, only to learn (from a post somewhere) about the undocumented or incorrect dual power cabling to the norco (4220) backplane. After recabling with a single molex to each backplane, the intermittent disk issues stopped. So I would not be surprised if this is a power issue with backplane. Due to the age of the 4220, norco no longer carries this revision of the backplane.

This is why I am searching for a new chasis, one that is more solidly built.
 
Gea,

Love napp-it as usual!
Question. When I download/activate 17.01pro the "pro monitor" stops working. When I run 16.08 it works fine. I poked through menus and such but can't seem to find if there's a toggle I'm missing.


Here is the info on my about page.
napp-it Appliance: PRO version (If a PRO version expires, functionality is reduced to the unlimited FREE version)

running on:SunOS san01 5.11 illumos-7fc68dd i86pc i386 i86pc
OpenIndiana Development oi_15118 X86 powered by illumos Copyright 2011 Oracle andor its affiliates All rights reserved Use is subject to license terms Assembled 19 February 2013

job/ task server : running
backgrnd-agents : arcstat, iostat, fsstat, nicstat, poolstat, prstat, zilstat
GUI accel-agents : disk, zfs, group, html
Z-RAID SSF agent : disabled

Any ideas? Perhaps this is an OI issue?
 
OI 151 a8 is VERY old. I do not run tests against but know that realtime monitoring over the websocket server does not work with newer napp-it releases due some incompatibilities with the old unpatched and insecure OpenSSL versions in OI.

napp-it from 2017 does not use the webocketserver any longer so it may work with an old OpenIndiana. Try:
- update to newest (17.03 dev)
- disable/enable monitoring ('mon') in the toplevel menu nearby logout

btw
Think about an update to a current OI Hipster or OmniOS (stable with regular bug/security fixes).
They also come with a newer ZFS with more features, SMB2 or other improvements.
 
Info from OmniTi (OmniOS)

If you are user of LTS (r151014) or Stable (r151020), please "pkg update" and be ready for a new BE and a reboot.

Included in this update are:

- bge fixes on HPa Gen9 systems
- MSI-X for NVMe is disablace on VMware (where MSI-X is a problem)
- mmap() now properly modifies a file's timestamps in ZFS.
- Default NFS server threads are now increased.
- i40e now uses multiple receive and transmit rings
- One mpt_sas panic bugfix

The install media has been updated too, as the bge fix will help HP Gen9 users. Install media pointers can be found here:

Installation


and for OmniOS bloody:

Because I'm experimenting with Kayak-for-ISO, I want to have fresh illumos-omnios bits (esp. Loader itself) on the public server. I've just pushed new packages (full build) out to the bloody repo. Update highlights include:

- AIO improvements in LX
- Some loader bugfixes
- blkdev fixes for devices with blksize > 512 bytes.
- illumos 7777 "Expose xdf minor nodes when in PV-HVM mode" fix from Delphix. We are taking this pre-integration because of its potential to help the new Kayak-based ISO installer.

I will not be cutting traditional install media, given I'm getting closer on the Kayak-for-ISO front. Existing users should "pkg update". New users should either install the old bits and upgrade, or take their chances on the WIP Kayak-for-ISO installer.

Watch this space for an updated Kayak-for-ISO, which will at least have a controlling-terminal fix in it (so you can do job control on shells, AND invoke format(1M) w/o it complaining about /dev/tty).


Happy updating!
Dan
 
Last edited:
OI 151 a8 is VERY old. I do not run tests against but know that realtime monitoring over the websocket server does not work with newer napp-it releases due some incompatibilities with the old unpatched and insecure OpenSSL versions in OI.

napp-it from 2017 does not use the webocketserver any longer so it may work with an old OpenIndiana. Try:
- update to newest (17.03 dev)
- disable/enable monitoring ('mon') in the toplevel menu nearby logout

btw
Think about an update to a current OI Hipster or OmniOS (stable with regular bug/security fixes).
They also come with a newer ZFS with more features, SMB2 or other improvements.

Thank you. I'll look at updating the OS then, I rather expected that was probably the issue.

I never updated the current OS through the package manager because when I did things would break (I can't recall now what exactly, but I did need to roll back the BE). Is OmniOS pretty good about being able to update the OS as I go or is it pretty common to need to refresh the install every now and then?

Thanks again.
 
Hello,

Is there a way to monitor fan speed or fan failures inside Napp-IT on a SuperMicro JBOD (CSE-837E26-RJBOD1) enclosure?
(in-band communication?)

Greetings,
Emile
 
Basically, you need to restore /var/web-gui/_log where the settings are.
On current napp-it, you can use the backup job to save them to a datapool.

With napp-it Pro ACL/complete you can restore them together with all user/idmapping/passwords etc in menu User
 
Basically, you need to restore /var/web-gui/_log where the settings are.
On current napp-it, you can use the backup job to save them to a datapool.

With napp-it Pro ACL/complete you can restore them together with all user/idmapping/passwords etc in menu User

I see it now, thank you!
 
Back
Top