OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

The basic Problem is that you can send a zfs stream to a file. You can then use that file as input for a zfs receive to restore a filesystem. But you cannot incremental update this file by an incremental zfs send process. While it may be possible to use other files for an incremental send, I would not declare this as practical.

It is also completely unsecure as a single wrong bit in a the file or one of the incrementals would make a restore impossible.

This is why the option to build a raid-z pool on encrypted or unencypted files and to backup these files is the best way to backup/restore a ZFS filesystem to a non-ZFS system - beside a filebased sync ex via rsync or a service provider that offers zfs directly (offering or allowing ZFS replication over the net to a ZFS filesystem )
 
Ok,

This is all starting to make sense now. Thank you Hammer and Gea.

So the files are encrypted. ZFS will know right away if one of the underlying files were to be corrupted and we are going with a RaidZ setup because it will give us the ability to resilver if one (or more) files were to be corrupted.

A few unknowns.

1. What kind of disk do you store your files on? Should I just make a pool where I store the files and not backup that pool. Basically this means that I need to have double the space of any data I am going to backup. If going this approach how to get data in the cloud incrementally?

2. Can you setup these files on something like S3FS or S3QL? It would be pretty cool if you could but I have no idea if that would work.

3. Do you setup a ZFS server on EC2 and have it run with its backing to these encrypted files and just manage its space as it grows.
 
I skimmed the first post in the thread. Lots of resources here! I would be interested in learning about and running OmniOS, but I was already planning on building with freenas. Is it worth it going the more "barebones" approach with OmniOS, or am I just as well off to go with freeNAS?
 
Ok,

This is all starting to make sense now. Thank you Hammer and Gea.

So the files are encrypted. ZFS will know right away if one of the underlying files were to be corrupted and we are going with a RaidZ setup because it will give us the ability to resilver if one (or more) files were to be corrupted.

A few unknowns.

1. What kind of disk do you store your files on? Should I just make a pool where I store the files and not backup that pool. Basically this means that I need to have double the space of any data I am going to backup. If going this approach how to get data in the cloud incrementally?

2. Can you setup these files on something like S3FS or S3QL? It would be pretty cool if you could but I have no idea if that would work.

3. Do you setup a ZFS server on EC2 and have it run with its backing to these encrypted files and just manage its space as it grows.

The method of a pool on encrypted files is only good for a "whole pool" backup for sensitive data.
If you simply want a local pool in sync with a remote storage via ZFS, you need a remote ZFS system.
AS I do not use this I cannot recommend a provider.
 
I skimmed the first post in the thread. Lots of resources here! I would be interested in learning about and running OmniOS, but I was already planning on building with freenas. Is it worth it going the more "barebones" approach with OmniOS, or am I just as well off to go with freeNAS?

All use cases and configurations on FreeNAS (BSD) or OmniOS (Solarish) are quite similar, does not matter if All-In-One under ESXi or barebone Storage,
 
Well Linux ZFS now supports native encryption. Seems that it's maturing.

This was shown at last OpenZFS conference and is based on the last state of OpenSolaris encryption. Oracle published their encryption based on this work as well. This means that real ZFS encryption as a property of a filesystem, developped on ZoL by Datto will be available in any Open-ZFS platform (BSD, Illumos, OSX, Linux) as a common feature as it does not rely on Linux GPL licenses.

No need for incompatible Geli, Luks or Lofi encryptions

http://open-zfs.org/wiki/Main_Page
 
Hello guys,

I have a very curious issue accessing napp-it gui via host name.
I have an AIO ESXI, ZFS with 2 network adapters. One of them in the 192.168.1.X class for general networking, and one is 172.16.1.2 use for iSCSI. On the ESXI I have a VMKERNEL port for iSCSI which is addressed 172.16.1.1.
Hostname of the zfs system is san.

Trouble is, when trying to access san hostname, it resolves to 172.16.1.2, I can't figure out how and why, and since there is no gateway, obviously I cant access the configuration.
I believe the issue started when migrating to Win10. However, pinging san during the first few seconds after a boot, resolves to the correct address 192.168.1.134.
After 10 seconds, resolves back to 172.16...
I'm utterly stumped. Device list on the router (which is the dns server) doesnt reveal anything unusual, and a Win10 VM running on ESXI resolves correctly.
Does anyone have an idea?

Thanks,

Update:

Ok, seems that disabling Link Layer Discovery and Mapper, then running:
  • ipconfig /flushdns
  • ipconfig /registerdns
  • ipconfig /release
  • ipconfig /renew
  • NETSH winsock reset catalog
  • NETSH int ipv4 reset reset.log
  • NETSH int ipv6 reset reset.log
And rebooting, reanbling the discovery and mapper solved the issue.
 
Last edited:
Hello Gea,
I hope you can assist me in a drive replacement I started yesterday.

I needed to increase my pool size without increasing my drive count (not enough chassis space).
I went and connected a new drive and used the replace command, re-silvering went fine and I powered off the system to remove the old drive.

I also switched the SATA connector for the new drive with the old one that was removed, and when I switched the system on, my pool is degraded.
My guess is it cannot find the new drive since I also switched the cables.

Is there a way to fix this without putting the drive in it's original SATA connector (it's a different controller as well).

Thanks !


Code:
NAME                       STATE     READ WRITE CKSUM      CAP            Product /napp-it   IOstat mess
    Pool1                      DEGRADED     0     0     0
      raidz1-0                 DEGRADED     0     0     0
        c3t50014EE602E3B705d0  ONLINE       0     0     0      3 TB           WDC WD30EFRX-68A   S:0 H:0 T:0
        c3t50014EE658385054d0  ONLINE       0     0     0      3 TB           WDC WD30EFRX-68A   S:0 H:0 T:0
        c3t50014EE65838F8A4d0  ONLINE       0     0     0      3 TB           WDC WD30EFRX-68A   S:0 H:0 T:0
        c6t1d0                 UNAVAIL      0     0     0  cannot open                       
      raidz1-1                 ONLINE       0     0     0
        c3t50014EE0037BEFD1d0  ONLINE       0     0     0      3 TB           WDC WD30EFRX-68A   S:0 H:0 T:0
        c3t50014EE0AE26BFB7d0  ONLINE       0     0     0      3 TB           WDC WD30EFRX-68A   S:0 H:0 T:0
        c3t50014EE0AE26BFCCd0  ONLINE       0     0     0      3 TB           WDC WD30EFRX-68A   S:0 H:0 T:0
        c3t50014EE0AE26C021d0  ONLINE       0     0     0      3 TB           WDC WD30EFRX-68A   S:0 H:0 T:0
      raidz1-2                 ONLINE       0     0     0
        c6t2d0                 ONLINE       0     0     0      5 TB           WDC WD50EFRX-68M   S:0 H:0 T:0
        c6t3d0                 ONLINE       0     0     0      5 TB           WDC WD50EFRX-68M   S:0 H:0 T:0
        c6t4d0                 ONLINE       0     0     0      5 TB           WDC WD50EFRX-68M   S:0 H:0 T:0
 
Hello Guys,

Another question about networking configuration.
Running OmniOS AIO.

I am trying to change the hostname.
I have set the nodename, changed the hosts file under /etc and /etc/inet
I have change the config/nodename under svc:/system/identity:node service, however every time I restart the service or reboot, the system reverts to the old hostname. The config/nodename doesnt revert to the old hostname, but the system insists on using the old hostname.

The only solution I found was creating a hostname.vmxnet3s0 file and setting the new hostname there.
That works, but it also wrecks havoc with the ip configuration. It creates another static ip address in addition to the DHCP address I created, and both are down until I delete and re-create the DHCP which is required every boot.

Does anyone have any idea what could be digging up the old host name from the abyss?

Thanks,
 
Hello Guys,

Another question about networking configuration.
Running OmniOS AIO.

I am trying to change the hostname.
I have set the nodename, changed the hosts file under /etc and /etc/inet
I have change the config/nodename under svc:/system/identity:node service, however every time I restart the service or reboot, the system reverts to the old hostname. The config/nodename doesnt revert to the old hostname, but the system insists on using the old hostname.

The only solution I found was creating a hostname.vmxnet3s0 file and setting the new hostname there.
That works, but it also wrecks havoc with the ip configuration. It creates another static ip address in addition to the DHCP address I created, and both are down until I delete and re-create the DHCP which is required every boot.

Does anyone have any idea what could be digging up the old host name from the abyss?

Thanks,

# do it in that order!!
# edit /etc/hosts -> newname
# exe hostname newname
# edit /etc/nodename

If you use napp-it
System > Network Eth > Hostname
 
Hello Gea,
I hope you can assist me in a drive replacement I started yesterday.

I needed to increase my pool size without increasing my drive count (not enough chassis space).
I went and connected a new drive and used the replace command, re-silvering went fine and I powered off the system to remove the old drive.

I also switched the SATA connector for the new drive with the old one that was removed, and when I switched the system on, my pool is degraded.
My guess is it cannot find the new drive since I also switched the cables.

Is there a way to fix this without putting the drive in it's original SATA connector (it's a different controller as well).

Thanks !


Code:
NAME                       STATE     READ WRITE CKSUM      CAP            Product /napp-it   IOstat mess
    Pool1                      DEGRADED     0     0     0
      raidz1-0                 DEGRADED     0     0     0
        c3t50014EE602E3B705d0  ONLINE       0     0     0      3 TB           WDC WD30EFRX-68A   S:0 H:0 T:0
        c3t50014EE658385054d0  ONLINE       0     0     0      3 TB           WDC WD30EFRX-68A   S:0 H:0 T:0
        c3t50014EE65838F8A4d0  ONLINE       0     0     0      3 TB           WDC WD30EFRX-68A   S:0 H:0 T:0
        c6t1d0                 UNAVAIL      0     0     0  cannot open                      
      raidz1-1                 ONLINE       0     0     0
        c3t50014EE0037BEFD1d0  ONLINE       0     0     0      3 TB           WDC WD30EFRX-68A   S:0 H:0 T:0
        c3t50014EE0AE26BFB7d0  ONLINE       0     0     0      3 TB           WDC WD30EFRX-68A   S:0 H:0 T:0
        c3t50014EE0AE26BFCCd0  ONLINE       0     0     0      3 TB           WDC WD30EFRX-68A   S:0 H:0 T:0
        c3t50014EE0AE26C021d0  ONLINE       0     0     0      3 TB           WDC WD30EFRX-68A   S:0 H:0 T:0
      raidz1-2                 ONLINE       0     0     0
        c6t2d0                 ONLINE       0     0     0      5 TB           WDC WD50EFRX-68M   S:0 H:0 T:0
        c6t3d0                 ONLINE       0     0     0      5 TB           WDC WD50EFRX-68M   S:0 H:0 T:0
        c6t4d0                 ONLINE       0     0     0      5 TB           WDC WD50EFRX-68M   S:0 H:0 T:0


Replacing disk by disk is ok but if you need to change the cable/port with port based detection like c6t2d0 you may need a reboot or import/export to load the pool from new controller ports.

WWN numbering like c3t50014EE0AE26C021d0 depend not on a controller port. This is a disk related number that keeps the same even on a new controller or server.
 
Replacing disk by disk is ok but if you need to change the cable/port with port based detection like c6t2d0 you may need a reboot or import/export to load the pool from new controller ports.

WWN numbering like c3t50014EE0AE26C021d0 depend not on a controller port. This is a disk related number that keeps the same even on a new controller or server.

Thanks, Export -> Import did the trick !
 
# do it in that order!!
# edit /etc/hosts -> newname
# exe hostname newname
# edit /etc/nodename

If you use napp-it
System > Network Eth > Hostname

Thanks.
Unfortunately it didnt work.
The moment I remove hostname.vmxnet3s0 which contains the new hostname, it defaults back to the old hostname.
On the following boot I get the addition to the hosts file from napp-it with the old hostname, although /etc/nodename and system identity remain with the new name.
I also tried commenting out the bootagent in napp-it to not add that section to the hosts file. It didnt add the section to the hosts file, but the system still grabs the old hostname from somwehre.
Completely stumped.

Any other ideas?
 
As this had never be a problem it must be related to one of your special settings like the hostname.nic (that is obsolete, now you use ipadm) or due booting into a different BE.
 
You can use Jobs > Backup to save napp-it settings, users, smbroups and idmappings to your datapool.
With napp-it Pro you can use menu Users > Resore to restore all or some settings.

Manually, you must save/restore /var/web-gui/_logs/*


btw from omnios-discuss
"Updates have been published for the following versions of OmniOS:

r151014 (LTS)
r151018 (Previous Stable)
r151020 (Current Stable)
r151021 (Bloody)

These updates provide minor security updates in tmpfs and procfs, and also adds support for NICs using the Chelsio Terminator 5 10/40Gb ASIC in the cxgbe driver. As these are updates to the kernel, they do require a reboot in order for the new BE to become active.

Per the usual, the new packages may be applied using: 'pkg update'

On behalf of the OmniOS team and OmniTI, I wish you a happy New Year!
/dale"


Happy New Year
 
The install on Linux is by no means "friendly" compared to OmniOS - I'm sure this will change in the future. It will be interesting to see what happens going forward with respect to what being the system of choice for ZFS.
 
Today I logged into the Napp-it interface for the first time in months. It was on version .9f ? and I upgraded to the lastest version. Unfortunately the web service has not come back up.
Running Open Indiana version 151a8
I did a ./etc/init.d/napp-it stop
When I run ./etc/init.d/napp-it start I get the following error:

ld.so.1: napp-it-mhttpd: fatal: libssl.so.1.0.0: open failed: No such file or directory
./napp-it: line 3: 9741 Killed /var/web-gui/data/tools/httpd/napp-it-mhttpd -c \*\*.pl -u napp-it -d /var/web-gui/data/wwwroot -p 81

Has anyone had this problem upgrading in the past?
Thanks for any help
 
I found libssl.so.1.0.0 in /opt/csw/lib so I copied it to /usr/lib but I'm still getting the same error code

root@nas:/opt/csw/bin# find / -name "libssl.so.1.0.0"
/opt/csw/lib/amd64/libssl.so.1.0.0
/opt/csw/lib/libssl.so.1.0.0
root@nas:/opt/csw/bin# cp /opt/csw/lib/libssl.so.1.0.0 /usr/lib/
 
OI 151a is VERY old. It has not got any serious update or all the security fixes around SSL.
You should really update to a current OmniOS stable or OI Hipster dev (I would prefer a reinstall)

But napp-it should work if you follow the instrctions at
http://napp-it.org/downloads/openindiana.html
 
Thanks Gea,
Its back running again! I want to move to OmniOS, but everything has been so stable I've had no reason to upgrade. My uptime is at 450+ days, and the only reason it's not more is because I added a 1TB SSD for my VM's. I'm planning on upgrading my storage this year when I have the funds. I would like to go to 2x 3tb mirror to a 4x 4TB z2 setup. I'll probably migrate to OmniOS then.
 
Can anyone point me to cheapest 10Gb card supported by OmniOS? Right now I'm looking at Intel X520's but I'm wondering if there is anything cheaper.

Thanks!
 
All other cheap 10G cards are EoL or badly or not supported.
Take the X520
 
I have an X520 in my OmniOS setup, and it works pretty well. No complaints whatsoever.


I did run into a really fun bug setting up my new 8TB HDDs with napp-it. Running parted -lm to list the disks out would hang and never complete. Using the disk initialize step wouldn't work either, because it would hang, presumably trying to do the same thing. I ended up finding a workaround by using format -e, picking the disk and using label to give it a basic partition setup. After that things worked out slick.

Also, thanks for including the TB/TiB conversions for volume creation and LUN/LU creation. This will be the first time I was able to create a pool, create a volume, and get COMSTAR to publish it out without it crapping itself due to one using TiB and the other using TB.
 
Gea, for LX zones. Just a thought. There should be an option to enter the zone configurations name. If I create /tank/zones/mymachine - I'd like to call the config mymachine not centos, ubuntu or debian. I also noticed that he shared folder is always created even if the Yes option is never selected.
 
Thanks for the suggestion.
Lightweight Linux container on top of OmniOS is a huge step to allow Linux services on Solarish.
The current support menu is more or less there to make you and me familiar with this.
 
This'll be considered a basic question. I have an solaris 11.3 nas using napp-it and everything works great. I recently setup an LACP on the machine, but afterwords lost outbound internet connection with the machine. It is still visible to all devices on the LAN which for security purposes probably is a better case, but I'd like to know why it did this. I'm guessing I did something wrong or missed a step. The LACP has been tested and I know it works, but I'd like to figure this out.

I followed the oracle documentation here: https://docs.oracle.com/cd/E23824_01/html/821-1458/fpjvl.html#gafxe "How to Create a Link Aggregation".

ifconfig -a output is attached as an image.
upload_2017-1-21_16-52-55.png


I noticed that the first connection is just completely odd, but did some reading that is the default. The second connection is the LACP with a static IP assigned to it. I'm not sure if I somehow set that as the IPV6 connection, it is missing IPV4, or what is going on. The LACP does work, but I'd like to figure out why it no longer has outbound connection for learning purposes. Thanks.
 
Boot mirror
You need a hardware raidcontroller if you want to mirror the ESXi bootdisk.
But as the intel 3700 is very reliable and a reinstall from Scratch (ESXi + napp-it from an ova template) is done in 20 min this is not really needed. Another option beside a hardwareraid is a 3,5" sata raid-1 enclosure for 2 x 2,5" disks or you can image your bootdisk example with clonezilla and use the second disk as cold spare.
I'm looking at this issue with a Lenovo TS440, which has limited expansion.

The x16 slot has a fat NIC. x4 will run an LSI2008 for passthru. That leaves an x1 & an old PCI slot, as well as 5x SATA ports from the chipset.

_Gea, can you recommend a specific SATA-RAID enclosure as described above?

Anyone know an x1 or PCI RAID1 card that supports email notifications under ESXi?

If not, I can live with the cold-spare scenario, but I'd like to make an informed decision.

Thanks, everybody.
 
There are some like a Raidsonic iR2420-2S-S2B or Icydock MB992SKR-B
that offer a hardware raid-1 enclosure for Sata.

They work quite well in case of a full disk failure.
But as this is not a ZFS Raid with checksums or with powerloss protection it can happen that both disks are not in sync ex on a outage during write (write hole problem) or because of any other disk problem on one disk. As such problems cannot be detected they can result in a damaged mirror/ boot partition.

This is why I use them widely but not as active raid but to hot-create a coldspare boot disk.
For this I create the mirror and remove the second disk after rebuild with a date note on it.
From time to time I reinsert the disks to update the mirror and remove it then again.
In case of a crash I have a working and up to date bootdisk

Overall I found that a reliable bootdisk is more worth that a mirror without a hardwareraid + BBU that you can trust more.
SSDs like an Intel enterprise S3510-80 with powerloss protection are more valuable like a cheap mirror of unreliable disks.
 
Thank you for the advice.

I'm leaning to a pair of used Intel DC SSDs & the cloning option, particularly as this is a small-budget setup.
 
I cannot confirm that.
I just tried to download the 17.01 ova and it worked for me.
 
I could not download the file while I was at the office. I am trying to do it now and its working, but kind of low (speed). Maybe there are some security issues on the way. I do not know. 2 hrs and I will have that file :)
 
The file is about 3,5GB.
Here in Germany (at home, different Internet provider than the source) i need less than 10 minutes
but the downöoad time depend on the connectivity between your and my provider.
 
ok, thanks for the info. did not know this is a private server. It actually took less, about 1 hour.
I have a question, however.

I have deployed the napp-IT-VM-template, basic configuration is done, but I cant see any disk.
I have VT-d enabled, passthrough on LSI HBA (Dell H200 flashed to IT-mode) in host management enabled and SCSI controller LSI parallel logic added in the napp-it VM.
What are possible mistakes I could have made?
update:
its unbelievable ;) I just did not have pressed my drives far enough while I was inserting them into the cage. Its all good now.
 
Last edited:
The ova template is not on a private server.
The 8 minutes is the download time from my local internet access provider to the different provider that hosts the file

btw.
The pass-through adapter is not the LSI parallel adapter. This is a virtual device.
You must add as pci-device

As it works now, you should have done this right.
 
Back
Top