OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,949
If this is napp-it on Solarish (OmniOS or Oracle Solaris)

If napp-it can see the disks within the VM, you can import the pool. Otherwise the pool is not available.

If this is napp-it on Linux, napp-it should see the disks as then napp-it runs native on Linux (with a reduced feature set compared to the Solaris/Unix version), see https://napp-it.org/downloads/linux_en.html
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,949
For a commercial cloud storage Amazon S3 is a de facto standard,

MinIO is an S3 compatible opensource cloud service, ultrafast and scalable nearly without limits and really easy to setup and maintain without the complexity of other cloud services. Basically the whole miniIO S3 server is a single file. Not the complexity of the solutions based on an Apache AMP stack.

As OmniOS has included minIO in the extra repository.
I have added support into napp-it.

You can now share a ZFS filesystem locally via NFS/SMB
and additionally for the Internet via S3 per ZFS filesystem.

Details, see https://forums.servethehome.com/ind...atible-client-server-minio.27524/#post-257208
 

Aesma

[H]ard|Gawd
Joined
Mar 24, 2010
Messages
1,850
Hello all.

I've been running nappit on OI for years.

I've started a replace of the wrong drive by mistake. Usually it wouldn't be a big issue, however I was already replacing another drive (expanding my pool) so now the result is a very slow resilver. The replaces aren't on the same vdev, there is no redundancy issue, I just want to avoid running the server for weeks or more just because of this mistake. Is there a way to cancel the replace ? I can't find it. I tried to offline the new disk, I've now physically removed it, but the "replacing" is still there, and the resilver is still slow as hell (about 100MiB/s instead of the usual 1,2GiB/s).

Code:
NAME                         STATE     READ WRITE CKSUM      CAP            Product /napp-it   IOstat mess
    abc                          DEGRADED     0     0     0
      raidz3-0                   DEGRADED     0     0     0
        c6t5000CCA24CE49B48d0    ONLINE       0     0     0      4 TB           HGST HDN724040AL   S:0 H:0 T:0
        replacing-1              DEGRADED     0     0     0
          c6t5000CCA24CE70C3Fd0  ONLINE       0     0     0      4 TB           HGST HDN724040AL   S:0 H:0 T:0
          c6t5000CCA23DC13FCEd0  UNAVAIL      0     0     0  cannot open                       
        c6t5000CCA24CE78517d0    ONLINE       0     0     0      4 TB           HGST HDN724040AL   S:0 H:0 T:0
        c6t5000CCA24CEA64DBd0    ONLINE       0     0     0      4 TB           HGST HDN724040AL   S:0 H:0 T:0
Thanks for your help
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,949
Unlike scrub there is no command to stop a resilver. As a resilver creates a mirror relation you can remove one disk of the mirror unless redundancy is not affected.

btw.
Current Illumos offer sorted/sequential resilver that is much faster than in former releases.
 

Aesma

[H]ard|Gawd
Joined
Mar 24, 2010
Messages
1,850
I can remove or I can't ? It's a raidz3 vdev with the original drive online so there shouldn't be a redundancy problem, right ?

For your other suggestion, would I need to upgrade the pool or just import it in a newer system ? I can't upgrade my OI, it doesn't work (I messed too much with it). My pool is at the latest Solaris compatible version so I'd rather not upgrade it.
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,949
In a Z3 you can loose up to three disks without a dataloss

About the running replace
c6t5000CCA24CE70C3Fd0 ONLINE

if you remove this disk, your resilver will stop with state=degraded.
Finally you must replace the faulted disk some time to go back to online state

If you are on ZFS v28 and want to go to current ZFS (v5000 with features):

-Install a new OS (can use a new bootdisk to keep the old one working) either OpenIndiana Hipster 2019.10 (with GUI) or OmniOS 151032 (minimalistic, suited for production use with a stable/lts and regular security fixes, free but optional with a commercial support option)

-Import the pool.

- Update pool to newest (napp-it click on pool version in menu Pools)
 

Aesma

[H]ard|Gawd
Joined
Mar 24, 2010
Messages
1,850
OK I'll remove that disk and start preparing to install Hipster. I use some windows apps with Wine and I need a GUI.

I want to stay on v28 though.

Thanks
 

Aesma

[H]ard|Gawd
Joined
Mar 24, 2010
Messages
1,850
So at first hipster looks like my old OI.

Then I see things are missing.

I'm having much trouble setting up vnc. I installed nappit and saw I could activate vnc with it, good, except where do I setup the password ?

edit : OK I installed again and didn't try to do vnc myself, just used nappit to activate it and it works perfect.
 
Last edited:

Aesma

[H]ard|Gawd
Joined
Mar 24, 2010
Messages
1,850
Now resilvering is set to last about 2 days (probably 3, from experience) so that's good.

I have a strange issue though. My pool had about 1% free space before I changed OS. That is, without reservation, that I had removed, as I need all the space I can get, and performance isn't my main concern. Now, after importing the pool, there is 0 free space, as in not a single bit. All the files seem to be there, but the free space is gone. I deleted a few files, and still, no free space. I vaguely remember having changed some parameter to gain a bit of space, but can't remember what. I don't think it's recordsize, and recordsize shouldn't affect free space anyway, right ?

Code:
NAME  PROPERTY              VALUE                  SOURCE
abc   type                  filesystem             -
abc   creation              Wed May  6 21:47 2015  -
abc   used                  161T                   -
abc   available             0                      -
abc   referenced            486K                   -
abc   compressratio         1.00x                  -
abc   mounted               yes                    -
abc   quota                 none                   default
abc   reservation           none                   default
abc   recordsize            128K                   default
abc   mountpoint            /abc                   local
abc   sharenfs              off                    default
abc   checksum              on                     default
abc   compression           off                    default
abc   atime                 on                     default
abc   devices               on                     default
abc   exec                  on                     default
abc   setuid                on                     default
abc   readonly              off                    default
abc   zoned                 off                    default
abc   snapdir               hidden                 default
abc   aclmode               discard                default
abc   aclinherit            restricted             default
abc   createtxg             1                      -
abc   canmount              on                     default
abc   xattr                 on                     default
abc   copies                1                      default
abc   version               5                      -
abc   utf8only              off                    -
abc   normalization         none                   -
abc   casesensitivity       sensitive              -
abc   vscan                 off                    default
abc   nbmand                off                    default
abc   sharesmb              off                    default
abc   refquota              none                   default
abc   refreservation        none                   default
abc   guid                  14230760374975883861   -
abc   primarycache          all                    default
abc   secondarycache        all                    default
abc   usedbysnapshots       0                      -
abc   usedbydataset         486K                   -
abc   usedbychildren        161T                   -
abc   usedbyrefreservation  0                      -
abc   logbias               latency                default
abc   dedup                 off                    default
abc   mlslabel              none                   default
abc   sync                  standard               default
abc   dnodesize             legacy                 default
abc   refcompressratio      1.00x                  -
abc   written               486K                   -
abc   logicalused           161T                   -
abc   logicalreferenced     16K                    -
abc   filesystem_limit      none                   default
abc   snapshot_limit        none                   default
abc   filesystem_count      none                   default
abc   snapshot_count        none                   default
abc   redundant_metadata    all                    default
abc   special_small_blocks  0                      default
abc   encryption            off                    default
abc   keylocation           none                   default
abc   keyformat             none                   default
abc   pbkdf2iters           0                      default
 
Last edited:

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,949
If there are no snaps, no reservations or zvols, the pool is simply full. Delete some files and associated snaps to regain space. You should enable compress. As it seems you have updated the pool to v5000, you can use the very efficient lz4 compress. This works only for newly created files ex when you move between filesystems that requires a copy.
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,949
I have just added support for Apple Time Machine via SMB into napp-it up from 19.12
(OmniOS, OI, kernelbased/ Solaris SMB server, see menu Services > Bonjour)
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,949
How does one go about upgrading to these weekly builds?

Thanks
A simple "pkg update" updates to newest state of the repository. The former state remains available as a bootenvironment. An update to a newer release ex 151030->151032 requires a change of the repository.
 

Nemesis_001

Weaksauce
Joined
Apr 3, 2011
Messages
69
A simple "pkg update" updates to newest state of the repository. The former state remains available as a bootenvironment. An update to a newer release ex 151030->151032 requires a change of the repository.
That's what I thought, then any idea why would my openssl version be 1.1.1.6?
 

Nemesis_001

Weaksauce
Joined
Apr 3, 2011
Messages
69
Ah, lol, I see it now. I guess it's easy to miss when your daughter is sick and wouldn't let you sleep all night.

Thanks.
 

Nemesis_001

Weaksauce
Joined
Apr 3, 2011
Messages
69
Cannot install TLS Perl modules or cannot mail?
Perl modules seem to have installed successfully.
However I still cannot enable tls email using the webgui.
Unencrypted email has also stopped working with Google smtp server after a while of working well.

The install is quite an old install, I think beginning with r151014.
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,949
At least a setup with a new and current OmniOS works without problems.

Possible Problems
A current Omnios require a new c compiler. If you only update, you do not have one. You can install a newer compiler via "pkg install gcc7". Then retry to install Perl modules.

Then try the menu Jobs > TLS eMail > TLS Test ex with a Google account

Onother problem can be that older OmniOS used Sun SSL while now OpenSSL is in use. This switch happened 151026->151028 with some problems with installed applications that use SSL.

If the problem persists I would do a clean reinstall of OmniOS. To keep napp-it settings, save/restore /var/web-gui/_log/* and readd users/groups with their uid/gid and smb groups with members.
 

Nemesis_001

Weaksauce
Joined
Apr 3, 2011
Messages
69
At least a setup with a new and current OmniOS works without problems.

Possible Problems
A current Omnios require a new c compiler. If you only update, you do not have one. You can install a newer compiler via "pkg install gcc7". Then retry to install Perl modules.

Then try the menu Jobs > TLS eMail > TLS Test ex with a Google account

Onother problem can be that older OmniOS used Sun SSL while now OpenSSL is in use. This switch happened 151026->151028 with some problems with installed applications that use SSL.

If the problem persists I would do a clean reinstall of OmniOS. To keep napp-it settings, save/restore /var/web-gui/_log/* and readd users/groups with their uid/gid and smb groups with members.
Works! Thanks.

It still shows up as TLS not supported, but it just works.
 

zeroARMY

[H]ard|Gawd
Joined
Feb 22, 2003
Messages
1,175
Has anyone attempted to make pretty dashboards for their zfs+napp-it systems using applications like Grafana, InfluxDB and Telegraf? I've seen some stunning looking dashboards on reddit and I'm getting jealous. There's no way I'm paying napp-it's ridiculous price for a visual monitoring extension for my little home server, when FreeNAS does it for free. I'll do the work to avoid that obnoxious fee.

Some examples of the best I've seen:
nmpphi8alet41.png

8qlgekwezvp41.png


Some guides I'll be looking at. They are very FreeBSD/FreeNAS and Linux focused, hopefully this won't be impossible on OpenSolaris...
https://technicalramblings.com/blog...b-and-telegraf-to-monitor-your-unraid-system/
https://thenewstack.io/how-to-setup-influxdb-telegraf-and-grafana-on-docker-part-1/
https://blog.linuxserver.io/2017/11...r-server-using-grafana-influxdb-and-telegraf/
https://techexpert.tips/grafana/grafana-monitoring-snmp-devices/
https://github.com/TheOnAndOnlyZenomat/Grafana-Setup/wiki
http://codedemigod.com/posts/monitoring-my-servers/
 
Last edited:

ARNiTECT

n00b
Joined
Aug 4, 2012
Messages
45
The CPU in one of my AiO servers limits me to ESXi 6.5U2.
After deploying the napp-it OVA template for ESXi 6.5 (napp-it-san026-may2018.ova) I upgraded from napp-it 18.01 to 19.12b7 and OmniOS v11 r151026c to 151032z
At 151028 I had to uninstall gcc6, which allowed me to complete the upgrade.
The setup guide (setup_napp-it_os.pdf) explains to "install gcc7 later"
Do I need to install a recent version of gcc such as gcc7?
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,949
You need a compiler like gcc7 only when you want to compile applications or ex tls support.
 

ARNiTECT

n00b
Joined
Aug 4, 2012
Messages
45
There is a new stable of OmniOS (free Solaris fork) v. 151034

New features since 151032 stable
https://github.com/omniosorg/omnios-build/blob/r151034/doc/ReleaseNotes.md
I have napp-it pro license, and will soon activate to get the pro releases. I am currently on napp-it 19.12b7 / omnios 151032z.
Should I wait until napp-it has been updated in line with OmniOS r151034, or activate license, download latest pro version, upgrade OmniOS to r151034, or some other order?
Regarding gcc & tls, if I understand this correctly, I'll need to consider this if I have problems setting up status emails.
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,949
Does not matter.
Any current napp-it free or Pro supports OmniOS 151030-34 incl TLS
 

aviat72

Limp Gawd
Joined
Dec 19, 2010
Messages
505
The WFH has given me some time to clean up the garage, and I uncovered my OpenIndiana Box which I set up about 8 years ago. Since things moved to the cloud, we stopped using it; but I want to revive it.

It uses an IBM branded (ServeRAID BR10i) LSI logic RAID card for which I had flashed the BIOS to use it as an HBA connected to five 2TB Seagate disks. The boot drive is an old SSD.

I got it to reboot after changing the PSU.

Problem is that I have no idea what the username or password are! Further after I get to the login screen, the keyboard seems to stop working (i.e. keystrokes dont echo when I try entering a username though it was working in the BIOS).

The data on the drives is not critical. The version of OpenIndiana is 1.51a.

Thoughts on how to proceed?

Given all the advances in the past eight years, which is the best OS to use if I am ok Reimaging

I am also building a new Ryzen 3600x box.

Can I run VMWare Workstation Player on it and run the zfs setup on the new Windows PC?

My current Mobo is an ASUS P8B WS C206 running an Intel Pentium G620 with 16GB of RAM.
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,949
I would just install either a current OpenIndiana Hipster 05.2020 or OmniOS stable 151034. You should then able to import the old pool (v.28) and update to newest pool version v5000 with new features like encryption.

If you want the graphical desktop use OpenIndiana otherwise OmniOS due the stable, long term stable and regular security and bug fixes.

https://omniosce.org/
https://napp-it.org/manuals/index_en.html (setup is 4.)

If you want to virtualize the NAS, use ESXi and pass-through for the HBA. The default for an old mainboard would be, download the .iso, burn a CD/DVD and install the OS. If the mainboard can boot from USB, you can use an USB stick with the OS installer.
 

aviat72

Limp Gawd
Joined
Dec 19, 2010
Messages
505
I would just install either a current OpenIndiana Hipster 05.2020 or OmniOS stable 151034. You should then able to import the old pool (v.28) and update to newest pool version v5000 with new features like encryption.

If you want the graphical desktop use OpenIndiana otherwise OmniOS due the stable, long term stable and regular security and bug fixes.

https://omniosce.org/
https://napp-it.org/manuals/index_en.html (setup is 4.)
Thanks!
If you want to virtualize the NAS, use ESXi and pass-through for the HBA. The default for an old mainboard would be, download the .iso, burn a CD/DVD and install the OS. If the mainboard can boot from USB, you can use an USB stick with the OS installer.
The other option I am considering is to use a VM on a Windows 10 Pro PC I am building. Goal is to reduce the number of PC boxes.

My current plan is to use the AMD 3600x with X570 board. I do not want to run Windows10 through ESXi since this will also be my son's gaming PC.

So I am thinking of using the VMWare Workstation Replay. Has anyone tried this setup (AMD Ryzen 3, X570, Windows 10 Pro + NAS on VMWare Workstation Player)
 

Captainquark

Weaksauce
Joined
Dec 14, 2011
Messages
99
Hi _Gea
Since my rig is quite old by now, it's time for something new. As I have two servers now (1 Esxi and 1 ZFS NAS), I am compelled by your idea of an AIO. I have read your manual on the napp-it website, but would like to run a few things by you, just to make sure I'm not running into problems later.
  1. I'll use a Samsung PM981a M.2 bootdisk (256GB) on a Supermicro X11SPi-TF that will host the storage VM. Do you see a problem in that? Unfortunately, the board has only one M.2 connector, so no mirror possible. Is it likely that this type of SSD will wear out quickly?
  2. So that storage VM will then have access to the data storage through my LSI 9300-8i and provide ZFS based storage to a CIFS data pool and Comstar or NFS for more VM's - correct?
Thanks for a short reassurance :)
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,949
1.
No problem. You use the bootdisk for ESXi and the storage VM.
Both produce a read based workload.

A mirror is mostly not needed.
If the bootdisk dies, you can reinstall ESXi and napp-it within a very short time and import the VMs.

2.
yes
 

Vengance_01

Supreme [H]ardness
Joined
Dec 23, 2001
Messages
6,194
Hi _Gea
Since my rig is quite old by now, it's time for something new. As I have two servers now (1 Esxi and 1 ZFS NAS), I am compelled by your idea of an AIO. I have read your manual on the napp-it website, but would like to run a few things by you, just to make sure I'm not running into problems later.
  1. I'll use a Samsung PM981a M.2 bootdisk (256GB) on a Supermicro X11SPi-TF that will host the storage VM. Do you see a problem in that? Unfortunately, the board has only one M.2 connector, so no mirror possible. Is it likely that this type of SSD will wear out quickly?
  2. So that storage VM will then have access to the data storage through my LSI 9300-8i and provide ZFS based storage to a CIFS data pool and Comstar or NFS for more VM's - correct?
Thanks for a short reassurance :)
Boot ESXi off of a small USB 3.0 stick. My current favorite are these sticks
https://www.amazon.com/dp/B07D7SX9NS/ref=twister_B07F718YLH?_encoding=UTF8&psc=1
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,949
I have added a small howto for backup/copy/sync local files from/to a napp-it ZFS filer from/to a cloudservice (or between cloudservices) like (Amazon) S3, Google or Microsoft.

http://www.napp-it.org/doc/downloads/cloudsync.pdf

update:
I have added configuration infos about https, encrypted files and setup for Amazon S3/minIO and Google drive
 
Last edited:

mmoole

n00b
Joined
Apr 4, 2012
Messages
5
Hello,

I need to buy new parts - at least mainboard, cpu and ram, because my very old X8SIL-F is dying.
This is a napp-in-one box with a flashed IBM1015 and is also used for some tinkering with various virtual machines from time to time, only for home use. So I don't need 10G networking or Intel Optane I guess.

On one hand I am quite on a limited budget, but on the other hand I don't want to buy stuff that is too old or not at least a bit future-proof...

So I looked around and if socket 2066 is the way to go, then the cheapest I found are
Supermicro X11SRM-F ca. 220€
Xeon W2102 ca. 230€ - can be upgraded later
32 or 64 GB DDR4 ECC RDIMM or LRDIMM ca. 130€ - lot's of offers, can be upgraded later
heatsink, fan, ca. 70€

But then I found questions:
* Supermicro lists on their site: 'Error Detection: Corrects single-bit errors' but nothing about double-bit errors - those where also corrected with the old board. Some other mainboards support this. Is that really an important thing?

* Did anybody have this board running with ESXI 6.5 or 6.7 and PCI passthrough? Maybe I was too dumb to use google for that, but I did only find on Supermicro's site that in the corresponding box on the matrix there is NO check mark . But although this does not mean a lot, I didn't find any report of this board working with ESXi.. and also no report of failing to do so ... has anybody a clue on this? In the manual it is stated that Intel VT-d is supported and can be set to enabled - am I missing something here?

* Is LGA2066 really the way to go? LGA3647 is way off looking at the prices. But would I be cheaper but still futureproof with LGA1151v2? But this seems to run with UDIMM only.?

By the way Timemachine on MacOS 10.15.5 was working OK (no need to tinker anything on MacOS side) with latest and upgraded OmniOS and napp-it, until the mainboard broke down :))

greetings

...EDIT
* or maybe AMD Epyc or Ryzen would be the more price-wise way to go?
 
Last edited:

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,949
Basically its all about number of pci-e lanes, possible RAM and number of cpu cores that are available. A single CPU mainboard like https://www.supermicro.com/en/products/motherboard/X11SPH-nCTF (board also available with SFP+ as nCTPF) comes with 12G SAS, 10G, 4 pci slots and allows up to 10 direct connected 4x NVMes and up to 28 cores (single cpu) and 1TB RAM. Price with all options is then in the range of a new car or a new Mac Pro.

2066 is newer and positioned sligtly below socket 3647. Socket 1151v2 is entry class with up to 128 GB RAM and a 6 core Xeon.

Basically its all about price and wanted expandability.

AMD has currently the better price/performance ratio but is more active in the gaming than the server sector. All my newer rendering machines are AMD. For a stable storage server I would still prefer an Intel board, best with Intel (10G if possible) nics and optionally a 12G SAS HBA with an LSI 3008 chipset.
 
Last edited:

shanester

Weaksauce
Joined
Mar 1, 2011
Messages
70
Running omnios-r151030-f1189fc02c
Napp-It 20.01a6 Pro

I am trying to modify smb settings to get timemachine via SMB to work.

When using the GUI to modify max_protocol property from "max_protocol=" to "max_protocol=3" nothing occurs. See below:
Current SMB properties: sharectl get smb

system_comment=
max_workers=1024
netbios_enable=true
netbios_scope=
lmauth_level=4
keep_alive=5400
wins_server_1=
wins_server_2=
wins_exclude=
signing_enabled=true
signing_required=false
restrict_anonymous=false
pdc=
ads_site=
ddns_enable=false
autohome_map=/etc
ipv6_enable=false
print_enable=false
traverse_mounts=true
map=
unmap=
disposition=
max_protocol=

When I run the command as root: sharectl set -p max_protocol=3
I get the following error:
Could not set property max_protocol: bad property value

The same error occurs if I try to add a property that doesn't exist, such as oplock_enable=true

I have looked for log files, such as /var/adm/messages that would help me figure out why this is occurring but cannot find them.

1. Where are the logs stored when modifications are made within napp-it (such as described above)? What about via cli?
2. Can someone tell me how to modify or add smb properties? sharectl set obviously isn't working for me.
 
Top