OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

Discussion in 'SSDs & Data Storage' started by _Gea, Dec 30, 2010.

  1. _Gea

    _Gea 2[H]4U

    Messages:
    3,855
    Joined:
    Dec 5, 2010
    OSX and its file system capabilities are based on HFS.
    Most but not all features are kept with extra .files (same filename) when you use "foreign" filesystems like fat or any other example via NFS/SMB shares.

    OSX hides the extra files. If you connect such a share via Windows or Unix/Linux you see the extra files
     
  2. _Gea

    _Gea 2[H]4U

    Messages:
    3,855
    Joined:
    Dec 5, 2010
    This depends not on Illumos but the question if you (can) compile all netatalk features.
    I use the ips repo at University Ulm for netatalk on Illumos, OmniOS Package Repository: uulm.mawi
    The netatalk there is without Spotlight support.

    Beside that (my personal opinion)
    AFP is EoL, Apple moved to SMB as default protocol
    Netatalk is EoL for the same reason, Netatalk - SerNet
    Beside that, netatalk was a pain because of Applle and all the incompatibilities with non HFS filesystems
     
  3. HammerSandwich

    HammerSandwich [H]ard|Gawd

    Messages:
    1,116
    Joined:
    Nov 18, 2004
    _Gea, this helps only with multiple clients, right?
     
  4. mkush

    mkush n00b

    Messages:
    36
    Joined:
    Aug 2, 2012
    So I played around with this further. I think the data the Finder is showing about the pics is NOT actually extended attributes since running xattr -l file.jpg returns no info. But still something odd is happening. Maybe it's just standard Finder behavior. Consider this sequence of events:

    1. File is on local HFS volume. Finder Get Info displays camera data.

    2. Copy file to OmniOS server. Finder displays no camera info. There is also no hidden ._file.jpg.

    3. The odd part. Copy the file back to the HFS volume and it AGAIN displays camera info, nothing was lost.

    So it seems that the camera info isn't extended attributes at all, it's just read from the metadata of the picture. But Finder only displays the info if the file is on an HFS volume (well actually local OpenZFS volumes also display it).

    BUT returning to actual extended attributes, by typing a comment into the Finder Get Info window or setting a color label on the file, you DO generate actual extended attributes. xattr -l file.jpg shows these. Copying this file to OmniOS, the comments and labels stay. And, this is the important part, there is still NO ._file.jpg. So I can only assume the attributes are being stored natively by ZFS.
     
  5. _Gea

    _Gea 2[H]4U

    Messages:
    3,855
    Joined:
    Dec 5, 2010
    or multiple tasks ex copy/ editings
     
  6. _Gea

    _Gea 2[H]4U

    Messages:
    3,855
    Joined:
    Dec 5, 2010
    There are no extended Mac attributes on ZFS/SMB beside the ._ files
    I suppose the ._ file is hidden.
    Do you have a Windows machine around. Windows shows the . files
     
  7. HammerSandwich

    HammerSandwich [H]ard|Gawd

    Messages:
    1,116
    Joined:
    Nov 18, 2004
    Ah, yes. 1 thread per stream. Thanks for clarifying.
     
  8. mkush

    mkush n00b

    Messages:
    36
    Joined:
    Aug 2, 2012
    I looked via a Windows VM. There are no ._ files visible. And are you saying that somehow, logged onto the OmniOS machine and doing an ls -a would not show the files?

    What there is, is a .DS_Store file created by Finder when it visits any folder. Via the Windows VM, I deleted that, on the theory that the extended attributes were coming from it. Then, without visiting the folder from Finder, I used Terminal to go there (cd /volumes/myshare/myfolder). Then I did xattr -l file.jpg, which STILL shows the extended attributes (a comment and color label that I added). Also ls -a showed no files beginning with a .

    So, unless OmniOS/ZFS/SMB is using some kind of crazy file hidden from ls -a and Windows, I can only conclude that the extended attributes are stored in the file itself. Am I missing something? Is there any way, on Illumos, to issue some command that would show the extended attributes of a file, or at least confirm their presence? That would verify that somehow they really are getting into the file itself.

    On the netatalk side of the fence, am I correct that an ea = sys statement in the config file WILL actually, for sure, result in the extended attributes being stored directly inside the files by ZFS?
     
  9. _Gea

    _Gea 2[H]4U

    Messages:
    3,855
    Joined:
    Dec 5, 2010
    I am not aware of an option to safe extended Mac attributes over SMB to ZFS.
    This is different to netatalk that can safe extended Mac attributes (ex via the ea setting)
     
  10. mkush

    mkush n00b

    Messages:
    36
    Joined:
    Aug 2, 2012
    Well, it sure LOOKS like it's saving them over SMB to ZFS. If anyone cares to know for sure, I'm happy to test further, but until someone suggests how to test or pokes a hole in what I've done, I'll have to conclude that it's working.
     
  11. mkush

    mkush n00b

    Messages:
    36
    Joined:
    Aug 2, 2012
    Sorry to keep going with this if nobody else is interested.

    I found that you can run ls -@ to confirm the presence of extended attributes in a file. If the file has them, instead of a + at the end of the permissions to indicate an ACL, it will print an @ only if extended attributes are present.

    I ran this test:

    1. Make a file on OmniOS server by vi test.txt.

    2. Run ls -@ file.txt on Omni, only + displayed.

    3. Do Get Info in Finder, or open the file in TextEdit, but make no changes.

    4. Run ls -@ file.txt on Omni, only + displayed.

    5. Set a Tag (color) in Finder, or put a comment in Get Info. I know tags and comments are stored as extended attributes on Macs.

    6. Run ls -@ file.txt on Omni, now @ is displayed indicating presence of extended attributes in the file.

    For step 5, I could also have edited the file on the Mac using TextEdit and the @ would have appeared. But, editing on the Mac using vi does not cause the @ to appear.

    Summary: ls -@ confirms presence of extended attributes on files only after they have metadata set in Finder or are edited with GUI tools.
     
  12. seijirou

    seijirou n00b

    Messages:
    34
    Joined:
    Feb 28, 2012
    Is it possible to turn off ACLs and just use linux style file permissions? ACLs are giving me loads of problems.
     
  13. mkush

    mkush n00b

    Messages:
    36
    Joined:
    Aug 2, 2012
    I tried that. Didn't work for me because if a file doesn't have ACLs, my Mac would invent them, and they were screwy. If the file already has legitimate ACLs the Mac respects them and all is good.

    What kinds of problems are you having? Once I got mine worked out, they work great.
     
  14. seijirou

    seijirou n00b

    Messages:
    34
    Joined:
    Feb 28, 2012
    I use my NAS as an NFS mount for various applications. Some of those applications (coincidentally all linux based) apparently want to chown and/or chmod files. The NAS is then interpreting those changes (I assume some interpretation) in to ACL changes. The application still works fine, however I then do a pool to pool rsync of those folders to a backup, and then finally have a different application (windows based using samba) read from those backups for an off-site cloud backup. That backup application looses read access because the permission tom-foolery that the application does trickles through to the backup copies. This is even though my samba share of the backup pool permits guest access; apparently ACLs take precedence.

    I used napp-it in the past in the same manner without any issues but back then it was just simple unix file permissions. I then dabbled with btrfs for a while and eventually came back to zfs but have discovered this issue that I didn't have before.

    I see a couple avenues to fix it, but haven't had luck on any of them.

    1. Assuming I'm correct about the ACLs taking precedence over samba guest access - just turn off ACLs.
    2. Configure the NFS clients to connect to NFS in a way that doesn't permit them to attempt permission changes on files
    3. Configure the NFS server in a way that doesn't permit clients to attempt permission changes on files.

    I don't know if any of these are possible, I've tried all 3 approaches without success. I'm all ears to other options too! Thanks.

    EDIT: I found a hacky solution that might work; adding the --chmod option to my pool-to-pool rsync. I'll test it.
     
    Last edited: Apr 26, 2016
  15. _Gea

    _Gea 2[H]4U

    Messages:
    3,855
    Joined:
    Dec 5, 2010
    Good news as this gives Mac users a better user experience.

    btw
    In current napp-it 16.04 you can just enable Bonjour/ multicast dns in Menu Services > Bonjour to connect SMB shares directly from a Mac with the nice Xserve icon instead the oldstyle SMB monitor icon with a Windows bluescreen.
     
  16. _Gea

    _Gea 2[H]4U

    Messages:
    3,855
    Joined:
    Dec 5, 2010
    You cannot disable ACL as the Solaris SMB server works like Windows: NTFS alike permissions only.
    What is your problem. ACLs are superior.
     
  17. seijirou

    seijirou n00b

    Messages:
    34
    Joined:
    Feb 28, 2012
    Multiple linux applications mount various directories via NFS and make chmod changes. A single offsite backup application needs to read everything.

    The linux applications setting chmod=xx0 turn in to ACLs that prevents the backup application connecting via samba from reading the directories. Whatever ACLs I set natively are squashed by the linux applications.

    ACLs might be superior in some circumstances, but in this circumstance they're just causing problems and I haven't found any way to reign them in and get them under control.
     
  18. seijirou

    seijirou n00b

    Messages:
    34
    Joined:
    Feb 28, 2012
    This might work, but unfortunately in napp-it when a job has an = sign, that seems to cut off the display of the full command in the UI. I checked and the full command is still there in the job's .par file but I don't have the best memory so 6 months down the line I'd forget what the command is if I can't just read in the gui and I certainly don't want to have to re-discover where those job files are located any time I want to make a change or I just want to refresh my memory.
     
  19. _Gea

    _Gea 2[H]4U

    Messages:
    3,855
    Joined:
    Dec 5, 2010
    There is indeed an incompatibility as
    - Solaris CIFS is Windows alike
    - NFS3 is Unix alike but without Authentication or Authorisation
    - rsync does not support Solaris NFS4 ACL at all

    What you can try
    - Select an everyone@=allow ACL with inheritance enabled
    - hinder NFS to modify permissions with aclmode=restricted
     
  20. seijirou

    seijirou n00b

    Messages:
    34
    Joined:
    Feb 28, 2012
    Thank you. Is aclmode=restricted set on the server or client side?

    EDIT: Oh I see, I think you're talking about in the filesystem. That's not an NFS option right?
     
  21. _Gea

    _Gea 2[H]4U

    Messages:
    3,855
    Joined:
    Dec 5, 2010
    This is a ZFS property
     
  22. seijirou

    seijirou n00b

    Messages:
    34
    Joined:
    Feb 28, 2012
    Understood, thank you.
     
  23. ldoodle

    ldoodle Limp Gawd

    Messages:
    161
    Joined:
    Jun 29, 2011
    Thanks. I've done my own reading and most reports do say it's not great. Especially when compared to ZFS. So i'll stick with ZFS (and no Windows at all).

    Can someone explain the read/write cache options with SSDs in ZFS. I'm going to get some (of the better ones) 128GB SSDs but I'm not sure on configuring them or how many to use for best results. I'll lose how ever many spinning disks as SSDs I use, which is fine, because I don't have port capacity.

    Is it one (or more) SSD per raid group, or per pool.
     
  24. _Gea

    _Gea 2[H]4U

    Messages:
    3,855
    Joined:
    Dec 5, 2010
    There are different caches and options with SSDs and ZFS.

    Write cache:
    This is always RAM based (a few seconds of write) and transforms small slow random writes to a single fast sequential write.

    Read cache
    The first readcache is rambased and called ARC. It works blockbased in a combination of last and most accessed blocks. You can extend the ARC with an SSD. This is called L2ARC. Such an L2ARC works on a per pool base.

    Slog device
    This is not a cache but a logdevice that logs all writes to a dedicated logdevice to allow fast writes over the regular rambased writecache to your pool. In case of a power outage, last writes can be restored from this logdevice. You need powerloss protection on the device ex ZeusRAM, Intel S3700 or Intel P750. Think of the Slog like a BBU + cache on a hardware raid. Similar - but faster.

    A logdevice can be quite small (10GB) but must must offer a very low latency and very high write iops. You must enable or allow sync write if you want to use the Slog. This is not a performance setting but a security option. Even with the fastest slog, writes are slower than with sync=disabled as you must write all data to your pool and additionally to the logdevice. A logdevice is also per pool but you can set sync on a per filesystem base.
     
    Last edited: May 3, 2016
    ldoodle likes this.
  25. mkush

    mkush n00b

    Messages:
    36
    Joined:
    Aug 2, 2012
    I've been running new OmniOS r151018 with kernel SMB2 since a day or two after it appeared. So far I'm pretty impressed. I'd still like Spotlight and finally did find that OpenCSW has a Netatalk package that installs Tracker too and is pre-configured for Spotlight. I'm testing it in a virtual machine. But SMB2 is so good and stable and everything that I'm not sure I'll ever use it. We'll see.

    This post is not about that however. The one issue I seem to have, using the Mac Finder, is that some operations are slower than I would expect. I'm running over 10GbE and find copying to and from the server to be quick. However, three things seem slow so far.

    1. Renaming a folder with a lot of stuff in it.
    2. Moving a folder with a lot of stuff in it.
    3. Deleting a folder with a lot of stuff in it.

    Folders with little content do all three operations quickly via Finder.

    Number 3 I can kind of understand, after all, there's more to delete. But it's still way slower than if it were happening locally, and way slower than I would expect, given the general speed of the server otherwise.

    Numbers 1 and 2 I don't understand at all. I can't see how the contents of the folder should effect renaming it, or even moving it. I can log onto the server by ssh and rename a folder and it happens instantaneously. Why slow via Finder? Does it maybe have to do with ACL inheritance (which I do have)?
     
  26. mkush

    mkush n00b

    Messages:
    36
    Joined:
    Aug 2, 2012
    Well, never mind I guess. 1 and 2 are immediate now (didn't test 3). Not sure what changed... Something on the Mac side I guess.
     
  27. davewolfs

    davewolfs Limp Gawd

    Messages:
    331
    Joined:
    Nov 7, 2006
    What release is suggested to be used these days with OmniOS is it time to upgrade from LTS?
     
  28. _Gea

    _Gea 2[H]4U

    Messages:
    3,855
    Joined:
    Dec 5, 2010
    If you only use NFS it does not really matter.
    If you use SMB you will find a much better performance due SMb 2.1 especially on a Mac or with 10G
    see http://napp-it.org/doc/downloads/performance_smb2.pdf

    Other improvements, see
    ReleaseNotes/r151018

    So yes, there are good reasons to upgrade to 151018, the current stable
     
  29. CopyRunStart

    CopyRunStart Limp Gawd

    Messages:
    153
    Joined:
    Apr 3, 2014
    Quick question on which drives are bad on this system.

    The "degraded" is next to the spare-1, spare-2, and spare-3. Are the spares degraded or the drives that say "unvailable"?

    And why does the pool still show degraded if the spares resilvered?


    Output of zpool status:

    NAME STATE READ WRITE CKSUM
    backup DEGRADED 0 0 0
    raidz2-0 DEGRADED 0 0 0
    c0t50014EE20B220BA4d0 ONLINE 0 0 0
    c0t50014EE20B237D95d0 ONLINE 0 0 0
    spare-2 DEGRADED 0 0 0
    c0t50014EE20B2550DAd0 UNAVAIL 46 61 0
    c0t50014EE2B5D2F865d0 ONLINE 0 0 0
    c0t50014EE20B271A01d0 ONLINE 0 0 0
    c0t50014EE20B277DD8d0 ONLINE 0 0 0
    c0t50014EE20B27A6C2d0 ONLINE 0 0 0
    c0t50014EE20B27ABE2d0 ONLINE 0 0 0
    c0t50014EE20B27C6C4d0 ONLINE 0 0 0
    c0t50014EE20B282A71d0 ONLINE 0 0 0
    c0t50014EE20B28397Ad0 ONLINE 0 0 1
    raidz2-1 ONLINE 0 0 0
    c0t50014EE20B28456Dd0 ONLINE 0 0 0
    c0t50014EE20B286057d0 ONLINE 0 0 0
    c0t50014EE20B28607Fd0 ONLINE 0 0 0
    c0t50014EE20B286F21d0 ONLINE 0 0 0
    c0t50014EE20B287593d0 ONLINE 0 0 0
    c0t50014EE260791674d0 ONLINE 0 0 0
    c0t50014EE260794440d0 ONLINE 0 0 0
    c0t50014EE2607B5CE5d0 ONLINE 0 0 0
    c0t50014EE2607C7F95d0 ONLINE 0 0 0
    c0t50014EE2607C7FD9d0 ONLINE 0 0 0
    raidz2-2 ONLINE 0 0 0
    c0t50014EE2607CBA76d0 ONLINE 0 0 0
    c0t50014EE2607CC79Cd0 ONLINE 0 0 0
    c0t50014EE2607CC7BFd0 ONLINE 0 0 0
    c0t50014EE2607CC7D5d0 ONLINE 0 0 0
    c0t50014EE2607CE1F6d0 ONLINE 0 0 0
    c0t50014EE2607D477Fd0 ONLINE 0 0 0
    c0t50014EE2607D4D92d0 ONLINE 0 0 0
    c0t50014EE2607D4DCCd0 ONLINE 0 0 0
    c0t50014EE2607D6630d0 ONLINE 0 0 0
    c0t50014EE2607DC913d0 ONLINE 0 0 0
    raidz2-3 DEGRADED 0 0 0
    c0t50014EE2607DDC11d0 ONLINE 0 0 0
    spare-1 ONLINE 0 0 1
    c0t50014EE2607DE23Cd0 ONLINE 0 0 0
    c0t50014EE2B5D28DC2d0 ONLINE 0 0 0
    c0t50014EE2B5CFC8FDd0 ONLINE 0 0 0
    spare-3 DEGRADED 0 0 0
    c0t50014EE2B5D198A3d0 UNAVAIL 133 170 0
    c0t50014EE2B5D2A589d0 ONLINE 0 0 0
    c0t50014EE2B5D21984d0 ONLINE 0 0 0
    c0t50014EE2B5D237ECd0 ONLINE 0 0 0
    c0t50014EE2B5D2507Dd0 ONLINE 0 0 0
    c0t50014EE2B5D272E9d0 ONLINE 0 0 0
    c0t50014EE2B5D27B4Ed0 ONLINE 0 0 0
    c0t50014EE2B5D28A77d0 ONLINE 0 0 0
    spares
    c0t50014EE2B5D28DC2d0 INUSE
    c0t50014EE2B5D2A589d0 INUSE
    c0t50014EE2B5D2F865d0 INUSE
    c0t50014EE2B5D358CFd0 AVAIL
    c0t50014EE2B5D3AF6Ed0 AVAIL
     
  30. _Gea

    _Gea 2[H]4U

    Messages:
    3,855
    Joined:
    Dec 5, 2010
    Only the disks marked as UNAVAIL are bad.
    The other messages may occur for example if if you have several short power outages where disks (pooldisks and spares) fail for a short time.

    What you should do now
    optionally: power off and check all cables and disk bays if they are inserted correctly, then

    - remove one AVAIL disks from the spares
    - replace then one UNAVAIL disk with this disk

    redo the same for the second UNAVAIL disk
    the active spares should then be AVAIL again

    - remove the bad UNAVAIL disk and check them (ex with a low level tool of the manufacturer)

    in raid-z2-3 you have two active spares but only one unavail disk
    If one active spare keeps online, try to remove, optionally try an export/import after a clear error

    - clear errors

    You should then be online again without bad disks or suspicious messages
     
  31. _Gea

    _Gea 2[H]4U

    Messages:
    3,855
    Joined:
    Dec 5, 2010
    napp-it as a 1/10G network switch

    I am just playing with the software bridge feature in OmniOS that allows to use Solaris/ OmniOS like a network switch.

    Example: If you have 4 nics in OmniOS, you can use one as usual as uplink (assign an ip) then bridge this nic with the others. When you connect clients to the other nics, they can see the server, each other and everything that is on the uplink port.

    Use cases now:
    - Some 1 or 10G nics on the server with some 1/10G clients (no extra 1/10G switch is needed).
    This will allow a silent and cheap high performance office storage server setup especially when paired with NVMe and SSDs.

    Near future!
    There are Illumos drivers on the way for the Intel X710 with 1 or 2 x QSFP+/ 8 x 10G per nic. With one of these cards (they are quite affordable) and a QSFP+ to QSFP+ or 2 QSFP+ to 8 x SFP+ Copper QSFP+ cables you should be able to create high performance 10G or 40G lab or video editing installations without a 10G/40G switch - with up to 8 x 10G or 2 x 40G clients per nic.


    You can enable bridging/ switching manually, see
    Bridging Overview - Oracle Solaris Administration: Network Interfaces and Network Virtualization

    I have added the bridge feature in napp-it Pro/2016.05 dev in menu System > Network Eth
    Comments especially about stability are welcome

    see http://www.napp-it.org/doc/downloads/napp-it.pdf
    page 43
     
    Last edited: May 15, 2016
  32. CopyRunStart

    CopyRunStart Limp Gawd

    Messages:
    153
    Joined:
    Apr 3, 2014
    Hey Gea, I'm unable to detach a disk from a pool and don't understand why. I'm trying to detach the first unavailable drive in this VDEV, which seems to have been replaced by spare-1. But when I try to detach it I get the following error:


    Code:
    zpool detach "backup" c0t50014EE2607DDC11d0
    
    oops...
    Could not proceed due to an error. Please try again later or ask your sysadmin.
    Maybe a reboot after power-off may help.
    
    cannot detach c0t50014EE2607DDC11d0: only applicable to mirror and replacing vdevs
    
    
    
    back/ return to former menu
    
    Code:
    raidz2-3                   DEGRADED     0     0     0
                c0t50014EE2607DDC11d0    UNAVAIL    131   129     0
                spare-1                  ONLINE       0     0     0
                  c0t50014EE2607DE23Cd0  ONLINE       0     0     0
                  c0t50014EE2B5D28DC2d0  ONLINE       0     0     0
                c0t50014EE2B5CFC8FDd0    ONLINE       0     0     0
                c0t50014EE20D474BECd0    ONLINE       0     0     0
                spare-4                  DEGRADED     0     0     0
                  c0t50014EE2B5D21984d0  UNAVAIL    133   131     0
                  c0t50014EE2B5D358CFd0  ONLINE       0     0     0
                c0t50014EE2B5D237ECd0    ONLINE       0     0     0
                c0t50014EE2B5D2507Dd0    ONLINE       0     0     0
                c0t50014EE2B5D272E9d0    ONLINE       0     0     0
                c0t50014EE2B5D27B4Ed0    ONLINE       0     0     0
                c0t50014EE2B5D28A77d0    ONLINE       0     0     0

    EDIT: Ok, realized I need to do zpool offline, not zpool detach. But after doing zpool offline on that disk, it still shows up in zpool status. What gives?
     
    Last edited: May 18, 2016
  33. _Gea

    _Gea 2[H]4U

    Messages:
    3,855
    Joined:
    Dec 5, 2010
    Try the napp-it menu
    Disks >> Remove on the unavail disk and one of the disks under spare-1
     
  34. ST3F

    ST3F Limp Gawd

    Messages:
    181
    Joined:
    Oct 19, 2011
    ... and delete errors ! :)
     
  35. _Gea

    _Gea 2[H]4U

    Messages:
    3,855
    Joined:
    Dec 5, 2010
    Today I got some new toys
    - A mainboard SuperMicro X10SDV-2C-7TP4F

    This board includes a 25W CPU (2C/4HT), 2 x Intel 10G SFP+, 2 x 1G and 16 x SAS/Sata (LSI 2116).
    You can add up to 128 GB RAM. I paid around 500 Euro for the board.
    The boards is also available with 8 or 16 cores.

    Then I received an Intel X710 QSFP+ (40G QSFP+ or 4 x SFP+) for around 400 Euro

    First impression:
    A perfect board for a ZFS storage and All-In-One (beside driver support for 10G)
    Currently I use the 1G nics

    The size of the Board is Flex ATX, a little smaller than uATX but it fits perfectly in a regular uATX/ATX case


    I have tried the following:
    Use the napp-it systemimage with a Intel S3510-80: ok
    Installation of OmniOS from USB stick: ok
    HBA 2116: ok, comes with IT mode firmware

    Intel X550: not yet supported
    Intel X710: not yet supported

    For both drivers are under development, see Illumos mailling list


    ESXi 6.0U2
    Installation: ok
    Pass-through of the LSI 2116: ok
    X550: not supported
    X710: not supported

    Hope that the drivers included soon or they can be installed seperately


    Napp-in-One
    Deploying the napp-it ZFS storage server template with pass-through: ok


    A short test with Solaris 11.3

    Setup from USB stick: fails
    Setup from Sata DVD: fails
    I have tried different bios settings regarding USB2/3 settings
     
    Last edited: May 23, 2016
  36. CopyRunStart

    CopyRunStart Limp Gawd

    Messages:
    153
    Joined:
    Apr 3, 2014
    If you get a chance can you try Solaris 11.3 on it?
     
  37. _Gea

    _Gea 2[H]4U

    Messages:
    3,855
    Joined:
    Dec 5, 2010
    Next week I am abroad.
    Maybe I can try this on monday
     
  38. CopyRunStart

    CopyRunStart Limp Gawd

    Messages:
    153
    Joined:
    Apr 3, 2014
    I'm having trouble understanding what is happening with a couple of my VDEVs. I don't fully understand which devices are resilvering, if the spares are actually active, and which devices are the spares.



    Code:
    raidz2-2                     DEGRADED     0     0 76.6M
                c0t50014EE2607CBA76d0      DEGRADED     0     0 76.6M
                spare-1                    DEGRADED     0     0     0
                  replacing-0              DEGRADED     0     0     0
                    c0t50014EE2607CC79Cd0  UNAVAIL      0     0     0
                    c0t50014EE2B7F1C6EAd0  DEGRADED     0     0     0  (resilvering)
                  c0t50014EE2B5D2A589d0    DEGRADED     0     0     0  (resilvering)
                c0t50014EE2607CC7BFd0      DEGRADED     0     0 76.6M
                c0t50014EE2607CC7D5d0      DEGRADED     0     0 76.5M
                c0t50014EE2607CE1F6d0      DEGRADED     0     0 76.6M
                c0t50014EE2607D477Fd0      UNAVAIL    131   175     0
                c0t50014EE2607D4D92d0      DEGRADED     0     0 76.6M
                c0t50014EE2607D4DCCd0      DEGRADED     0     0 76.5M
                c0t50014EE2607D6630d0      DEGRADED     0     0 76.6M
                spare-9                    DEGRADED     0     0     0
                  c0t50014EE2607DC913d0    UNAVAIL      0     0     0
                  c0t50014EE2B5D2F865d0    DEGRADED     0     0     0  (resilvering)
              raidz2-3                     DEGRADED     0     0     0
                replacing-0                DEGRADED     0     0     0
                  c0t50014EE2607DDC11d0    UNAVAIL      0     0     0
                  c0t50014EE20D479E32d0    DEGRADED     0     0     0  (resilvering)
                spare-1                    ONLINE       0     0     0
                  c0t50014EE2607DE23Cd0    ONLINE       0     0     0
                  c0t50014EE2B5D28DC2d0    ONLINE       0     0     0
                c0t50014EE2B5CFC8FDd0      ONLINE       0     0     0
                c0t50014EE20D474BECd0      ONLINE       0     0     0
                c0t50014EE2B5D358CFd0      ONLINE       0     0     0
                c0t50014EE2B5D237ECd0      ONLINE       0     0     0
                c0t50014EE2B5D2507Dd0      ONLINE       0     0     0
                c0t50014EE2B5D272E9d0      ONLINE       0     0     0
                c0t50014EE2B5D27B4Ed0      ONLINE       0     0     0
                c0t50014EE2B5D28A77d0      ONLINE       0     0     0
            spares
              c0t50014EE2B5D28DC2d0        INUSE
              c0t50014EE2B5D2A589d0        INUSE
              c0t50014EE2B5D2F865d0        INUSE
    
    What is spare-1? And which drive is it replacing?
     
  39. _Gea

    _Gea 2[H]4U

    Messages:
    3,855
    Joined:
    Dec 5, 2010
    You must check the indentation starting from a vdev like raid z2-3

    the following is a replacing with two disks
    Code:
    -> raid z2-3
        ->replacing 0 (ignore naming, check only following replacements for this "disk position")
           c0.. unavail
           c0.. resilvering
       ->spare-1 (this is also a "disk position" with two following replacements)
           c0.. unavail
           c00  resilvering
       -> c0.. (disk ok)
    
    replacing -0 and spare-1 means that there are two events where a failed disk is beeing replaced by another one. Ignore the naming. See it as at this position of the vdev a disk fails and another tries to take the place.

    Such a confusing situation also regarding namings can happen if you have a short power failure or a cable disconnect where a disk fails. In this case a spare jumps in. On next occurance the next spare jump in and so on. This interferes with the former faulted disks come back.

    This is a reason why you may not activate all disks as hot spare. One is enough. Activate the others or replace manually on problems.

    What you can do now
    - wait until all replaces are finished then manually remove a disk from such a pair where both are online or remove the unavail pair member
    - or power down and check cabing and backplanes first. After power on a rebuild will start again
     
    Last edited: May 21, 2016
  40. rob-fantini

    rob-fantini n00b

    Messages:
    2
    Joined:
    May 19, 2016
    I ran into the same issue. the solution here was:


    cd /usr/bin

    mv perl perl-

    ln -s ../perl5/5.16.1/bin/i386/perl

    mv /var/web-gui/data/napp-it/CGI/auto/IO /var/web-gui/data/napp-it/CGI/auto/IO-