OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

Discussion in 'SSDs & Data Storage' started by _Gea, Dec 30, 2010.

  1. jad0083

    jad0083 Limp Gawd

    Messages:
    130
    Joined:
    Apr 30, 2006
    Yes. you should be able to flash it with an LSI 9211-8i IT firmware, to fully present the disks to the system (which ZFS prefers). With that said, that hba should be able to give you an additional 8 SATA/SAS hdds via 2x breakout cables, or even more via expanders.
     
  2. Nemesis_001

    Nemesis_001 n00bie

    Messages:
    49
    Joined:
    Apr 3, 2011
    Apologies, I mean 2 file systems on one pool which resides on one vdev.

    Meanwhile, the damn 8TB drives are stuck in customs.
     
  3. jad0083

    jad0083 Limp Gawd

    Messages:
    130
    Joined:
    Apr 30, 2006
    hey Gea,

    When you did your testing on optane as an slog, were you able to do any comparative testing on the different scsi drivers on esxi (lsi sas, pvscsi, nvme)?
     
  4. _Gea

    _Gea 2[H]4U

    Messages:
    3,751
    Joined:
    Dec 5, 2010
    I have used the default virtual disk driver (lsi)
     
  5. jad0083

    jad0083 Limp Gawd

    Messages:
    130
    Joined:
    Apr 30, 2006
  6. _Gea

    _Gea 2[H]4U

    Messages:
    3,751
    Joined:
    Dec 5, 2010
    new in napp-it 18.06 Dev (Apr 11)

    to be prepared for next OmniOS 151026 stable in may (or current bloody)
    https://github.com/omniosorg/omnios-build/blob/r151026/doc/ReleaseNotes.md

    Support for vdev removal (new ZFS feature) in menu Pools > Shrink Pool (OmniOS 151025+)
    Support for poolwide checkpoints (new ZFS feature) in menu Snaps > Checkpoint (OmniOS 151025+) and Pool > Import

    Disk Detection: adds ATTO HBAs (ATTO, a media specialist now supports Illumos)
    Disk Map: correct detection of HBA even if disks on different LSI mptsas HBAs shows the same c-number ex c0t500..
    Disk Map: add dd disk detection
    Disk Location: dd detection for all disks
     
    Last edited: Apr 11, 2018
  7. _Gea

    _Gea 2[H]4U

    Messages:
    3,751
    Joined:
    Dec 5, 2010
    The new OmniOS is the first Open-ZFS storage distribution to include the vdev remove (Pool shrink).
    Oracle Solaris 11.4 also comes with this feature but it seems with less restrictions.

    Open-ZFS, at least currently lacks the support for a vdev remove of a basic or mirror vdev when a raid-z [1-3] vdev is part of the pool or a remove raid-Z[1-3] at all or add a raid-z [1-3] after a remove of ex a basic/mirror vdev what limits its use cases. Support of raid-Z [2-3] is expected in Open-ZFS (but not Z1), Bug #7614: zfs device evacuation/removal - illumos gate - illumos.org

    Open-ZFS ex OmniOS that is the first to include this feature also requires a re-mapping table with a continous small RAM need/reservation and small performance degration.This is listed in the output of a zpool status. A manual zpool remap can fix this.

    It seems that Solaris 11.4 does not have these restrictions
    vdev removal, poolwide checkpoints/snaps or imp... | Oracle Community
     
    Last edited: Apr 12, 2018
  8. _Gea

    _Gea 2[H]4U

    Messages:
    3,751
    Joined:
    Dec 5, 2010
    Oracle Solaris 11.4 Beta Refresh is now available for download:
    https://community.oracle.com/message/14784683#14784683

    This refresh includes new capabilities and additional bug fixes.
    Some new features in this release:

    • ZFS vdev removal (all vdev types incl Raid-Zn)
    • ZFS Scheduled Scrub
    • SMB 3.1.1
     
    Last edited: Apr 28, 2018
  9. _Gea

    _Gea 2[H]4U

    Messages:
    3,751
    Joined:
    Dec 5, 2010
    A new snapshot of Openindiana Hipster 2018.04 is available

    OpenIndiana Hipster is a rolling distribution of the opensource Solaris fork Illumos with a snapshot every 6 months. It comes in the three flavours GUI with a Mate desktop, Text (very similar to OmniOS, another Illumos distribution) and Minimal.

    https://wiki.openindiana.org/oi/2018.04+Release+notes
     
  10. Pocatello

    Pocatello DC Moderator and [H]ard DCOTM x3 Staff Member

    Messages:
    6,033
    Joined:
    Jun 15, 2005
    I would like to create one of these but I don't know where to start. The user guide in the first post appears to be outdated.

    What software and OS is recommended?

    I would love to create a system using my current hardware which is older and outdated, but it can be given this new task. Once I get comfortable with the new OS and software I would like to build a second system with super-fast as my goal.

    Thanks for this thread!
     
  11. _Gea

    _Gea 2[H]4U

    Messages:
    3,751
    Joined:
    Dec 5, 2010
    OpenIndiana follows the idea of the former OpenSolaris as you can install a pure server version and a desktop capable version with GUI, many desktop apps and with a very wide use case range. OpenIndiana is a pure community project without a commercial support option.

    The main alternative is OmniOS with a single use case, ZFS storage server. It includes all what you need for that with a beta, stable and long term stable together with a commercial support option. There is no GUI option and the repository is strictly reduced to storage with a strong focus minimalism for a production quality stability.

    Setup, see 3.) at http://napp-it.org/manuals/index_en.html
     
  12. _Gea

    _Gea 2[H]4U

    Messages:
    3,751
    Joined:
    Dec 5, 2010
    OmniOS 151026 (may.07.2018) is out

    Release note: https://github.com/omniosorg/omnios-build/blob/r151026/doc/ReleaseNotes.md
    Download: https://downloads.omniosce.org/media/r151026/

    Main improvements:

    - Protection against the Meltdown Intel CPU vulnerability announced earlier this year
    - Experimental support for bhyve - a fast, lightweight and modern hypervisor
    - Sparse-branded zones, clocking in under 4MB per zone
    - An improved Installer which is dramatically faster making the OmniOS installation procedure
    one of the fastest in the industry. The new installer also provides many more options for
    customising the installed system.
    - A new lightweight default MTA (Dragonfly Mail Agent)
    - Fault management improvements for SSD disks

    ZFS features
    - Improved support for ZFS pool recovery (import good data from a damaged pool)
    - The new zfs remove of toplevel vdevs
    - zfs checkpoint features (poolwide checkpoint to make even a fs destroy/ vdev add/remove undoable)
    - Support for raidz2 and raidz3 boot disks

    Hardware
    - support of the new BroadCom tri-mode HBAs

    napp-it supports OmniOS 151026 up from 18.01 (apr.02)
    http://napp-it.org/downloads/changelog_en.html

    Update, see 3.)
    http://napp-it.org/manuals/index_en.html
     
    Last edited: May 8, 2018
  13. _Gea

    _Gea 2[H]4U

    Messages:
    3,751
    Joined:
    Dec 5, 2010
    ToddW2 likes this.
  14. CopyRunStart

    CopyRunStart Limp Gawd

    Messages:
    150
    Joined:
    Apr 3, 2014
    Hey Gea is it normal for the disks to show up twice when using Multi-Path? 2 HBAs, each with one cable connected to the JBOD.

    How do I add these disks to a pool? Which do I select of the two?

    EDIT: Fixed by adding my drive vendor and model to /etc/driver/drv/scsi_vhci.conf.
     
    Last edited: May 16, 2018
  15. gcooper

    gcooper n00bie

    Messages:
    6
    Joined:
    Feb 22, 2014
    I hope it is appropriate to post this here. My apologies if not.

    On OmniOS v11 r151016 and Napp-IT 0.9f6, I have three legitimately and physically removed drives displaying as "Removed". All pools are ONLINE (with new drives).

    How can I eliminate the removed drives from the display? A reboot? Trying to avoid that...

    Thanks in advance,

    G
     
  16. _Gea

    _Gea 2[H]4U

    Messages:
    3,751
    Joined:
    Dec 5, 2010
    The disks overview is based on iostat.

    Iostat keeps an inventory of disks and errors since bootup so remembering removed disks is as expected and as intended.
    This behaviour is essential if a disks suddenly fails and you need to know where it is or to check errors since last bootup.

    There are (unsupported) hacks to delete entries but in general only a bootup deletes the iostat memory.
     
    gcooper likes this.
  17. gcooper

    gcooper n00bie

    Messages:
    6
    Joined:
    Feb 22, 2014
    Perfect explanation. Thank you. G
     
  18. Nemesis_001

    Nemesis_001 n00bie

    Messages:
    49
    Joined:
    Apr 3, 2011
    Hi,

    Any reason to upgrade from ESXI 6.5u2 to ESXI 6.7?
     
  19. Ja'Man

    Ja'Man n00bie

    Messages:
    22
    Joined:
    Nov 10, 2011
    Gea,

    Hi, i have been using Napp-it for 6 or 7 years now and have installed on maybe a dozen NAS;s, most of them production and all have ran mostly without issue, at least without any issues due to napp-it. Thank you very much for all your great work!!!

    2 related questions. I would like the ability to ftp remotely to one of them, using some sort of NAT transversing vpn (softether vpn, etc), or anything that will allow me to ftp to it WITHOUT opening any ports on the firewall for the NAS;s network. Do you have any experience or knowledge of anyone doing so or a good guide for it?! And second question, I was able to get proftpd working, but can only authenticate using anonymous, if I restrict it to a certain user the authentication fails. Any suggestions or a good guide for getting that to work?! Thanks again!!
     
  20. _Gea

    _Gea 2[H]4U

    Messages:
    3,751
    Joined:
    Dec 5, 2010
    For ftp (or a more secure option sftp (ex with WinSCP as client) you can user port/ip forwarding on your internet gateway or
    you can use a VPN tunnel from your internet gateway to securely connect a remote host or network
    For sftp you can use local users and you do not need a 3rd party package

    Regarding proftp, see https://napp-it.org/extensions/proftpd_en.html
     
  21. Nemesis_001

    Nemesis_001 n00bie

    Messages:
    49
    Joined:
    Apr 3, 2011
    Hi,

    I have unified my 2 RAID-Z1 arrays on 8 small disks to a RAID-Z2 on 4 8TB drives.
    It was easy as pie, all the SMB and NFS shares were working once booted up, and the iSCSI LUN remained identical.
    The ZFS system never ceases to impress me.

    Anyway, the 8TB drives are much louder than anything I had before.
    I have an OmniOS VM on ESXI and a Win10 VM to handle media services.

    The WIN10 metadata and vmx files are stored on an NFS share, and the OS drive is passed using iSCSI RDM.
    Since default behavior of NFS is to always SYNC write which is awfully slow, I configured this way so atleast the metadata would be on NFS and written safely, while the OS data is not SYNC so much faster.

    When the Win10 OS is online, Every 3-4 seconds there is a short seek on the datapool, which is pretty loud and distracting. I don't see any disk writes on the Win10 resource monitor, at least not that frequent.
    When the Win10 OS is offline there are no writes.

    Any idea how to find out what's causing it and perhaps maybe cache the data for a while and not write that often?

    P.S.
    I have a 32GB Optane laying around. Based on this use pattern, any advantage to using the optane as SLOG or L2ARC?
    The server is mostly used for Plex, SMB sharing for Kodi, Backup storage for other PCs and downloading from Usenet.

    Thanks,
     
  22. 2hforum

    2hforum n00bie

    Messages:
    1
    Joined:
    Jul 11, 2018
    ZFS failing disks may cause system unresponsive or zpool hangs

    Oracle support doc (ID 1316531.1, https://support.oracle.com/knowledge/Sun Microsystems/1316513_1.html) seems to indicate that there might be a big problem in some (all?) OpenSolaris-based ZFS storage systems when vdevs (e.g. HDDs, SSDs) do not fail completely.

    What to do about this?
     
  23. _Gea

    _Gea 2[H]4U

    Messages:
    3,751
    Joined:
    Dec 5, 2010
    I have seen hanging pools with semidead disks in Illumos based systems (OpenZFS). What I do is usually reduce disk timeout from a default 60s to say 15s. The 7s of TLER disks seems too low for larger pools.

    But if you really want an answer, ask at Illumos discuss or dev where developpers are around
    https://wiki.illumos.org/display/illumos/illumos+Mailing+Lists
     
  24. _Gea

    _Gea 2[H]4U

    Messages:
    3,751
    Joined:
    Dec 5, 2010
    It may be the rambased writecache of ZFS with a flush every few seconds or an indexing service in Windows.

    regarding the Optane
    While the 32GB Optane is much slower than the bigger Optanes I would use it as ZIL, optionally partition it and use 10 GB for Slog and the rest for L2Arc (Optane is ok with concurrent use)
     
  25. Nemesis_001

    Nemesis_001 n00bie

    Messages:
    49
    Joined:
    Apr 3, 2011
  26. dedobot

    dedobot [H]Lite

    Messages:
    75
    Joined:
    Jun 19, 2012
    Ha just for the curious:
    Years ago I build a a storage rig for my an close friend's drone/video hobby.Second hand Intel PL5000 MB, 2x IBM m1015 , 2 RAIDz2 vdevs/pools, 8 2TB HDDS per vdev. Solaris 11.1 + nappit. Last month his UPS died and he ditched it. Last week was a storm, power stroke and the Intel MB and the redundant PSU fried. I changed the MB with an ASUS desktop one. Solaris booted w/o any problems, then we discovered that one of the IBM 1015 is dead too. But when we swapped the HDDs - the vdev and data are completely fine. Impressive rock solid combo Solaris+zfs. Solaris runs even better at the ASUS MB :). Of course there is no ECC RAM, so I disabled scrubs and put the vdevs in read mode until I build a new rig for him with OmniOS - he want to start small business and Solaris licence is unobtainable due to schizophrenic cost. If there was perpetual licence for something like 1k USD - the guy will paid it. But it's 1500USD annually :) .
     
  27. _Gea

    _Gea 2[H]4U

    Messages:
    3,751
    Joined:
    Dec 5, 2010
    For a single socket server Solaris is less than 900 USD/year but with OmniOS there is a stable/ long term stable OpenSource Solaris fork even with a commercial support option if wanted.
     
  28. _Gea

    _Gea 2[H]4U

    Messages:
    3,751
    Joined:
    Dec 5, 2010
  29. Nemesis_001

    Nemesis_001 n00bie

    Messages:
    49
    Joined:
    Apr 3, 2011
    Hi Gea,

    Tried passing the Optane directly to Omni-OS, didnt work. Afterwards I saw you mention it wouldnt work in the manual.
    Tried then creating vdisks and attaching to an NVME Controller. They show up with the correct capacity, but removed.

    Any idea what's wrong?

    Thanks
     
  30. _Gea

    _Gea 2[H]4U

    Messages:
    3,751
    Joined:
    Dec 5, 2010
    Is this a 32 GB Optane (seen troubles with it) or an 800P/900P (works in my setups with ESXi 6.7) ?
    Does it work with an LSI SAS controller in ESXi ?
     
  31. mikeo

    mikeo Limp Gawd

    Messages:
    178
    Joined:
    May 17, 2006
    So it's been a long time since I've messed with this, is it possible yet to do pci passthrough on a lsi raid sas card from a windows host to a virtualized linux guest OS and have the guest os run all the ZFS stuff? Or is ESXi still the only way to do that. Planning on a threadripper 32 core build this fall and would be nice to combine my zfs server and desktop into one box.
     
  32. _Gea

    _Gea 2[H]4U

    Messages:
    3,751
    Joined:
    Dec 5, 2010
    ESXi is quite the the smallest Type-1 Hypervisor with best guest OS support for Linux, Unix and Windows and with perfectly working pass-through for LSI HBAs so the best way to seperate VM server, Storage server and guest OSs, still my nr 1 AiO environment.

    On Windows Unix support is not as good and pass-through of hardware seems possible up from Server 2016 but I would not say rhis is a good base for AiO with a full featured storage appliance.

    Linux (Proxmox) offers ZFS as a possible filesystem so you can use ZFS directly. This is similar to SmartOS, a virtualisation/cloud environment with a Solarish base and quite native ZFS. But on both you cannot add a full featured storage appliance like FreeNAS (based on Free-BSD) or my napp-it (based on Solaris or the free Solaris fork OI or OpenIndiana) so you must manage storage mostly via CLI
     
    mikeo likes this.
  33. mikeo

    mikeo Limp Gawd

    Messages:
    178
    Joined:
    May 17, 2006
    Yeah, guess I'll just stick with my openindiana server managing it via CLI.
     
  34. Nemesis_001

    Nemesis_001 n00bie

    Messages:
    49
    Joined:
    Apr 3, 2011
    Yes, it's the 32GB optane, running ESXI 6.5. Right now running on a virtual SAS controller instead of NVME.
    The LSI SAS HBA passes through without any issue.

    P.S.
    I moved the Windows VM files to the optane, so the Raid-Z2 is only being used as iSCSI LUN for the Windows VM. Something is still always writing to the Raid-Z2 array, and I cannot figure out why. It's quite annoying.
    Even when the machine is in sleep state, and CPU usage is 0, there are still writes every few seconds until I power off the VM. Really scratching my head what could it be.

    Also, several cautionary tales of lessons learned during this change to anyone interested:

    1. Do not connect a PCI-E x1 device to PCI-E slots coming from the CPU. Most CPUs (Xeon E3, Core i3-i5 etc) support x16, x8 and x4. Connect it only to a slot coming from the PCH.
    The LSI SAS HBA connected to x8 slot also coming from the CPU starting spewing out tons of errors and OmniOS moved it to a faulted state and the entire array was gone.
    I had to move the Optane x1 device to a PCH based slot, and then dig in the manual to get the SAS card out of faulted state.
    The array went back online and was fine afterwards.

    2. Do not change the size of a LOG device or Cache device that are attached to an array in the Hypervisor. I initially assigned all of my Optane to partitions used for LOG and Cache. Then I wanted to move one of the VM files to also be stored on the Optane, so I wont get Hypervisor writes to the RaidZ array.
    I was pretty sure I removed the devices from the array / pool, but I probably only removed one. The subsequent OmniOS boot, I got a faulty array with an Assert: ZPOOL_STATUS_OK. It's quite an obscure error and couldnt find much information about the likely cause. It took quite a while to figure out. Had to remove the cache and log devices from the VM, export the array, re-import it and then remove the "missing" log device. I was sweating profusely during this time.
    The pool imported back intact.
     
  35. _Gea

    _Gea 2[H]4U

    Messages:
    3,751
    Joined:
    Dec 5, 2010
    mikeo likes this.
  36. _Gea

    _Gea 2[H]4U

    Messages:
    3,751
    Joined:
    Dec 5, 2010
    Midnight Commander
    Midnight commander (mc) is a filebrowser that allows browsing/ copying/ moving/ viewing and editing files from a text console.

    As mc is not in the current Solaris 11.4 or OmniOS repository the fastest way to get it is my online installer:
    wget -O - www.napp-it.org/midnight_commander | perl

    If midnight commander does not show correct borders in Putty:
    - open Putty settings Window > Translation: modify ex from UTF-8 to ISO 8859-1 (Latin-1, West Europe)
    - reconnect
     
  37. mikeo

    mikeo Limp Gawd

    Messages:
    178
    Joined:
    May 17, 2006
    Switched from openindiana to Ubuntu, imported the pools and my new asus xg-c100c 10g cards worked in ubuntu from the get go. Occasionally they time out between windows 10 and ubuntu, not sure why that is happening yet. Installed napp-it on Linux and it's running great!

    Thanks _Gea!
     
  38. _Gea

    _Gea 2[H]4U

    Messages:
    3,751
    Joined:
    Dec 5, 2010
  39. _Gea

    _Gea 2[H]4U

    Messages:
    3,751
    Joined:
    Dec 5, 2010
    AiO with Solaris 11.4 on ESXi 6.7
    No support from current Vmware tools for Solaris 11.4 final from today
    vmware-tools / open-vm-tools on 11.4b |Oracle Community

    My findings/ "just a hack"

    VMware vmtools for Solaris from ESXi 6.7
    Executing on a textonly setup of S11.4 final on ESXi 6.7

    Installer vmware-install-pl on 11.4 installs but fails with a message
    Package "SUNWuiu8" not found when executing vmtool/bin/vmware-config-tools.pl

    This can be skipped by editing vmtool/bin/vmware-config-tools.pl in line 13026
    when you comment out the check for SUNWuiu8

    When you then run vmtool\bin\vmware-config-tools.pl it hangs due a missing /usr/bin/isalist
    I copied isalist over from a Solaris 11.3, made it executable and then vmware-config-tools.pl works

    After a reboot I got the message vmools installed with a console message
    Warning: Signature verification of module /kernel/drv/amd64/vmmemctl failed

    same with verification of the vmxnet3s driver
    vmxnet3s reports deprecated "misc/mac"

    Not sure if this is critical

    vmxnet3s and guest restart from ESXi works

    Gea
     
  40. _Gea

    _Gea 2[H]4U

    Messages:
    3,751
    Joined:
    Dec 5, 2010
    Full OS/ Appliance disaster recovery

    On current napp-it 18.09dev I have added a new function/menu: System > Recovery
    with modified replication jobs to make recovery easy. The idea behind is:

    To recover a fully configured appliance from a BE (bootenvironment):

    1. backup current BE: create a replication job (require 18.09dev) with current BE as source

    2. reinstall OS and napp-it

    3. restore BE: create a replication job with the BE as source and rpool/ROOT as target (require 18.09dev)

    4. activate the restored BE and reboot

    This BE backup/restore can be done also manually via replication and zfs send