OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

Yes. you should be able to flash it with an LSI 9211-8i IT firmware, to fully present the disks to the system (which ZFS prefers). With that said, that hba should be able to give you an additional 8 SATA/SAS hdds via 2x breakout cables, or even more via expanders.
 
hey Gea,

When you did your testing on optane as an slog, were you able to do any comparative testing on the different scsi drivers on esxi (lsi sas, pvscsi, nvme)?
 
new in napp-it 18.06 Dev (Apr 11)

to be prepared for next OmniOS 151026 stable in may (or current bloody)
https://github.com/omniosorg/omnios-build/blob/r151026/doc/ReleaseNotes.md

Support for vdev removal (new ZFS feature) in menu Pools > Shrink Pool (OmniOS 151025+)
Support for poolwide checkpoints (new ZFS feature) in menu Snaps > Checkpoint (OmniOS 151025+) and Pool > Import

Disk Detection: adds ATTO HBAs (ATTO, a media specialist now supports Illumos)
Disk Map: correct detection of HBA even if disks on different LSI mptsas HBAs shows the same c-number ex c0t500..
Disk Map: add dd disk detection
Disk Location: dd detection for all disks
 
Last edited:
The new OmniOS is the first Open-ZFS storage distribution to include the vdev remove (Pool shrink).
Oracle Solaris 11.4 also comes with this feature but it seems with less restrictions.

Open-ZFS, at least currently lacks the support for a vdev remove of a basic or mirror vdev when a raid-z [1-3] vdev is part of the pool or a remove raid-Z[1-3] at all or add a raid-z [1-3] after a remove of ex a basic/mirror vdev what limits its use cases. Support of raid-Z [2-3] is expected in Open-ZFS (but not Z1), Bug #7614: zfs device evacuation/removal - illumos gate - illumos.org

Open-ZFS ex OmniOS that is the first to include this feature also requires a re-mapping table with a continous small RAM need/reservation and small performance degration.This is listed in the output of a zpool status. A manual zpool remap can fix this.

It seems that Solaris 11.4 does not have these restrictions
vdev removal, poolwide checkpoints/snaps or imp... | Oracle Community
 
Last edited:
A new snapshot of Openindiana Hipster 2018.04 is available

OpenIndiana Hipster is a rolling distribution of the opensource Solaris fork Illumos with a snapshot every 6 months. It comes in the three flavours GUI with a Mate desktop, Text (very similar to OmniOS, another Illumos distribution) and Minimal.

https://wiki.openindiana.org/oi/2018.04+Release+notes
 
I would like to create one of these but I don't know where to start. The user guide in the first post appears to be outdated.

What software and OS is recommended?

I would love to create a system using my current hardware which is older and outdated, but it can be given this new task. Once I get comfortable with the new OS and software I would like to build a second system with super-fast as my goal.

Thanks for this thread!
 
OpenIndiana follows the idea of the former OpenSolaris as you can install a pure server version and a desktop capable version with GUI, many desktop apps and with a very wide use case range. OpenIndiana is a pure community project without a commercial support option.

The main alternative is OmniOS with a single use case, ZFS storage server. It includes all what you need for that with a beta, stable and long term stable together with a commercial support option. There is no GUI option and the repository is strictly reduced to storage with a strong focus minimalism for a production quality stability.

Setup, see 3.) at http://napp-it.org/manuals/index_en.html
 
OmniOS 151026 (may.07.2018) is out

Release note: https://github.com/omniosorg/omnios-build/blob/r151026/doc/ReleaseNotes.md
Download: https://downloads.omniosce.org/media/r151026/

Main improvements:

- Protection against the Meltdown Intel CPU vulnerability announced earlier this year
- Experimental support for bhyve - a fast, lightweight and modern hypervisor
- Sparse-branded zones, clocking in under 4MB per zone
- An improved Installer which is dramatically faster making the OmniOS installation procedure
one of the fastest in the industry. The new installer also provides many more options for
customising the installed system.
- A new lightweight default MTA (Dragonfly Mail Agent)
- Fault management improvements for SSD disks

ZFS features
- Improved support for ZFS pool recovery (import good data from a damaged pool)
- The new zfs remove of toplevel vdevs
- zfs checkpoint features (poolwide checkpoint to make even a fs destroy/ vdev add/remove undoable)
- Support for raidz2 and raidz3 boot disks

Hardware
- support of the new BroadCom tri-mode HBAs

napp-it supports OmniOS 151026 up from 18.01 (apr.02)
http://napp-it.org/downloads/changelog_en.html

Update, see 3.)
http://napp-it.org/manuals/index_en.html
 
Last edited:
Hey Gea is it normal for the disks to show up twice when using Multi-Path? 2 HBAs, each with one cable connected to the JBOD.

How do I add these disks to a pool? Which do I select of the two?

EDIT: Fixed by adding my drive vendor and model to /etc/driver/drv/scsi_vhci.conf.
 
Last edited:
I hope it is appropriate to post this here. My apologies if not.

On OmniOS v11 r151016 and Napp-IT 0.9f6, I have three legitimately and physically removed drives displaying as "Removed". All pools are ONLINE (with new drives).

How can I eliminate the removed drives from the display? A reboot? Trying to avoid that...

Thanks in advance,

G
 
The disks overview is based on iostat.

Iostat keeps an inventory of disks and errors since bootup so remembering removed disks is as expected and as intended.
This behaviour is essential if a disks suddenly fails and you need to know where it is or to check errors since last bootup.

There are (unsupported) hacks to delete entries but in general only a bootup deletes the iostat memory.
 
Gea,

Hi, i have been using Napp-it for 6 or 7 years now and have installed on maybe a dozen NAS;s, most of them production and all have ran mostly without issue, at least without any issues due to napp-it. Thank you very much for all your great work!!!

2 related questions. I would like the ability to ftp remotely to one of them, using some sort of NAT transversing vpn (softether vpn, etc), or anything that will allow me to ftp to it WITHOUT opening any ports on the firewall for the NAS;s network. Do you have any experience or knowledge of anyone doing so or a good guide for it?! And second question, I was able to get proftpd working, but can only authenticate using anonymous, if I restrict it to a certain user the authentication fails. Any suggestions or a good guide for getting that to work?! Thanks again!!
 
For ftp (or a more secure option sftp (ex with WinSCP as client) you can user port/ip forwarding on your internet gateway or
you can use a VPN tunnel from your internet gateway to securely connect a remote host or network
For sftp you can use local users and you do not need a 3rd party package

Regarding proftp, see https://napp-it.org/extensions/proftpd_en.html
 
Hi,

I have unified my 2 RAID-Z1 arrays on 8 small disks to a RAID-Z2 on 4 8TB drives.
It was easy as pie, all the SMB and NFS shares were working once booted up, and the iSCSI LUN remained identical.
The ZFS system never ceases to impress me.

Anyway, the 8TB drives are much louder than anything I had before.
I have an OmniOS VM on ESXI and a Win10 VM to handle media services.

The WIN10 metadata and vmx files are stored on an NFS share, and the OS drive is passed using iSCSI RDM.
Since default behavior of NFS is to always SYNC write which is awfully slow, I configured this way so atleast the metadata would be on NFS and written safely, while the OS data is not SYNC so much faster.

When the Win10 OS is online, Every 3-4 seconds there is a short seek on the datapool, which is pretty loud and distracting. I don't see any disk writes on the Win10 resource monitor, at least not that frequent.
When the Win10 OS is offline there are no writes.

Any idea how to find out what's causing it and perhaps maybe cache the data for a while and not write that often?

P.S.
I have a 32GB Optane laying around. Based on this use pattern, any advantage to using the optane as SLOG or L2ARC?
The server is mostly used for Plex, SMB sharing for Kodi, Backup storage for other PCs and downloading from Usenet.

Thanks,
 
Hi,

I have unified my 2 RAID-Z1 arrays on 8 small disks to a RAID-Z2 on 4 8TB drives.
It was easy as pie, all the SMB and NFS shares were working once booted up, and the iSCSI LUN remained identical.
The ZFS system never ceases to impress me.

Anyway, the 8TB drives are much louder than anything I had before.
I have an OmniOS VM on ESXI and a Win10 VM to handle media services.

The WIN10 metadata and vmx files are stored on an NFS share, and the OS drive is passed using iSCSI RDM.
Since default behavior of NFS is to always SYNC write which is awfully slow, I configured this way so atleast the metadata would be on NFS and written safely, while the OS data is not SYNC so much faster.

When the Win10 OS is online, Every 3-4 seconds there is a short seek on the datapool, which is pretty loud and distracting. I don't see any disk writes on the Win10 resource monitor, at least not that frequent.
When the Win10 OS is offline there are no writes.

Any idea how to find out what's causing it and perhaps maybe cache the data for a while and not write that often?

P.S.
I have a 32GB Optane laying around. Based on this use pattern, any advantage to using the optane as SLOG or L2ARC?
The server is mostly used for Plex, SMB sharing for Kodi, Backup storage for other PCs and downloading from Usenet.

Thanks,

It may be the rambased writecache of ZFS with a flush every few seconds or an indexing service in Windows.

regarding the Optane
While the 32GB Optane is much slower than the bigger Optanes I would use it as ZIL, optionally partition it and use 10 GB for Slog and the rest for L2Arc (Optane is ok with concurrent use)
 
Ha just for the curious:
Years ago I build a a storage rig for my an close friend's drone/video hobby.Second hand Intel PL5000 MB, 2x IBM m1015 , 2 RAIDz2 vdevs/pools, 8 2TB HDDS per vdev. Solaris 11.1 + nappit. Last month his UPS died and he ditched it. Last week was a storm, power stroke and the Intel MB and the redundant PSU fried. I changed the MB with an ASUS desktop one. Solaris booted w/o any problems, then we discovered that one of the IBM 1015 is dead too. But when we swapped the HDDs - the vdev and data are completely fine. Impressive rock solid combo Solaris+zfs. Solaris runs even better at the ASUS MB :). Of course there is no ECC RAM, so I disabled scrubs and put the vdevs in read mode until I build a new rig for him with OmniOS - he want to start small business and Solaris licence is unobtainable due to schizophrenic cost. If there was perpetual licence for something like 1k USD - the guy will paid it. But it's 1500USD annually :) .
 
For a single socket server Solaris is less than 900 USD/year but with OmniOS there is a stable/ long term stable OpenSource Solaris fork even with a commercial support option if wanted.
 
It may be the rambased writecache of ZFS with a flush every few seconds or an indexing service in Windows.

regarding the Optane
While the 32GB Optane is much slower than the bigger Optanes I would use it as ZIL, optionally partition it and use 10 GB for Slog and the rest for L2Arc (Optane is ok with concurrent use)

Hi Gea,

Tried passing the Optane directly to Omni-OS, didnt work. Afterwards I saw you mention it wouldnt work in the manual.
Tried then creating vdisks and attaching to an NVME Controller. They show up with the correct capacity, but removed.

Any idea what's wrong?

Thanks
 
Is this a 32 GB Optane (seen troubles with it) or an 800P/900P (works in my setups with ESXi 6.7) ?
Does it work with an LSI SAS controller in ESXi ?
 
So it's been a long time since I've messed with this, is it possible yet to do pci passthrough on a lsi raid sas card from a windows host to a virtualized linux guest OS and have the guest os run all the ZFS stuff? Or is ESXi still the only way to do that. Planning on a threadripper 32 core build this fall and would be nice to combine my zfs server and desktop into one box.
 
ESXi is quite the the smallest Type-1 Hypervisor with best guest OS support for Linux, Unix and Windows and with perfectly working pass-through for LSI HBAs so the best way to seperate VM server, Storage server and guest OSs, still my nr 1 AiO environment.

On Windows Unix support is not as good and pass-through of hardware seems possible up from Server 2016 but I would not say rhis is a good base for AiO with a full featured storage appliance.

Linux (Proxmox) offers ZFS as a possible filesystem so you can use ZFS directly. This is similar to SmartOS, a virtualisation/cloud environment with a Solarish base and quite native ZFS. But on both you cannot add a full featured storage appliance like FreeNAS (based on Free-BSD) or my napp-it (based on Solaris or the free Solaris fork OI or OpenIndiana) so you must manage storage mostly via CLI
 
  • Like
Reactions: mikeo
like this
Is this a 32 GB Optane (seen troubles with it) or an 800P/900P (works in my setups with ESXi 6.7) ?
Does it work with an LSI SAS controller in ESXi ?

Yes, it's the 32GB optane, running ESXI 6.5. Right now running on a virtual SAS controller instead of NVME.
The LSI SAS HBA passes through without any issue.

P.S.
I moved the Windows VM files to the optane, so the Raid-Z2 is only being used as iSCSI LUN for the Windows VM. Something is still always writing to the Raid-Z2 array, and I cannot figure out why. It's quite annoying.
Even when the machine is in sleep state, and CPU usage is 0, there are still writes every few seconds until I power off the VM. Really scratching my head what could it be.

Also, several cautionary tales of lessons learned during this change to anyone interested:

1. Do not connect a PCI-E x1 device to PCI-E slots coming from the CPU. Most CPUs (Xeon E3, Core i3-i5 etc) support x16, x8 and x4. Connect it only to a slot coming from the PCH.
The LSI SAS HBA connected to x8 slot also coming from the CPU starting spewing out tons of errors and OmniOS moved it to a faulted state and the entire array was gone.
I had to move the Optane x1 device to a PCH based slot, and then dig in the manual to get the SAS card out of faulted state.
The array went back online and was fine afterwards.

2. Do not change the size of a LOG device or Cache device that are attached to an array in the Hypervisor. I initially assigned all of my Optane to partitions used for LOG and Cache. Then I wanted to move one of the VM files to also be stored on the Optane, so I wont get Hypervisor writes to the RaidZ array.
I was pretty sure I removed the devices from the array / pool, but I probably only removed one. The subsequent OmniOS boot, I got a faulty array with an Assert: ZPOOL_STATUS_OK. It's quite an obscure error and couldnt find much information about the likely cause. It took quite a while to figure out. Had to remove the cache and log devices from the VM, export the array, re-import it and then remove the "missing" log device. I was sweating profusely during this time.
The pool imported back intact.
 
Midnight Commander
Midnight commander (mc) is a filebrowser that allows browsing/ copying/ moving/ viewing and editing files from a text console.

As mc is not in the current Solaris 11.4 or OmniOS repository the fastest way to get it is my online installer:
wget -O - www.napp-it.org/midnight_commander | perl

If midnight commander does not show correct borders in Putty:
- open Putty settings Window > Translation: modify ex from UTF-8 to ISO 8859-1 (Latin-1, West Europe)
- reconnect
 
Switched from openindiana to Ubuntu, imported the pools and my new asus xg-c100c 10g cards worked in ubuntu from the get go. Occasionally they time out between windows 10 and ubuntu, not sure why that is happening yet. Installed napp-it on Linux and it's running great!

Thanks _Gea!
 
AiO with Solaris 11.4 on ESXi 6.7
No support from current Vmware tools for Solaris 11.4 final from today
vmware-tools / open-vm-tools on 11.4b |Oracle Community

My findings/ "just a hack"

VMware vmtools for Solaris from ESXi 6.7
Executing on a textonly setup of S11.4 final on ESXi 6.7

Installer vmware-install-pl on 11.4 installs but fails with a message
Package "SUNWuiu8" not found when executing vmtool/bin/vmware-config-tools.pl

This can be skipped by editing vmtool/bin/vmware-config-tools.pl in line 13026
when you comment out the check for SUNWuiu8

When you then run vmtool\bin\vmware-config-tools.pl it hangs due a missing /usr/bin/isalist
I copied isalist over from a Solaris 11.3, made it executable and then vmware-config-tools.pl works

After a reboot I got the message vmools installed with a console message
Warning: Signature verification of module /kernel/drv/amd64/vmmemctl failed

same with verification of the vmxnet3s driver
vmxnet3s reports deprecated "misc/mac"

Not sure if this is critical

vmxnet3s and guest restart from ESXi works

Gea
 
Full OS/ Appliance disaster recovery

On current napp-it 18.09dev I have added a new function/menu: System > Recovery
with modified replication jobs to make recovery easy. The idea behind is:

To recover a fully configured appliance from a BE (bootenvironment):

1. backup current BE: create a replication job (require 18.09dev) with current BE as source

2. reinstall OS and napp-it

3. restore BE: create a replication job with the BE as source and rpool/ROOT as target (require 18.09dev)

4. activate the restored BE and reboot

This BE backup/restore can be done also manually via replication and zfs send
 
Back
Top