OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

Discussion in 'SSDs & Data Storage' started by _Gea, Dec 30, 2010.

  1. _Gea

    _Gea 2[H]4U

    Messages:
    3,668
    Joined:
    Dec 5, 2010
    ZFS is crash resistent.
    This means that ZFS - a Copy on Write Filesystem is not corrupt after a crash during writes (unlike older filesystems). This does not mean that a VM or a database is consistent on a crash as ZFS uses up to 4 GB RAM as writecache for a better performance. Think of an accounting software where you put off an amount from one account and the system crashes prior you can add it to another account (money in data nirwana as the ramcache is lost on a crash).

    ZFS offers sync write, a mechanism where every commited write is logged. On a crash the commited writes are done on next reboot to allow a database or VM to be consistent. Sadly sync write requires a log device with powerloss protection and ultra low latency for a good performance. In the past these log devices (slog) were expensive and despire slow compared to fast writes without sync.

    The new Intel Optane is a game-changing technology. When you use them as an Slog, even sequential sync writes are nearly as fast as writes without sync. If you use Optane not for Slog but the pool itself, it opens a new performance level on small random writes and sequential writes. Even a filer with sync enabled is possible now. I am impressed!

    See http://napp-it.org/doc/downloads/optane_slog_pool_performane.pdf
     
  2. _Gea

    _Gea 2[H]4U

    Messages:
    3,668
    Joined:
    Dec 5, 2010
  3. stevebaynet

    stevebaynet Limp Gawd

    Messages:
    204
    Joined:
    Nov 9, 2011
    Hey Gea,

    Just got an alert that one of my rpool disks died. laaaaaame. However, when i login the web UI is unresponsive. If i recall previously, you said it is depending on certain commands completed, one of them being format.

    When I login to the console and execute format, it just says "Searching for disks.." and hangs.

    I'm running OmniOS latest and greatest. What next? do i need to find the disk first and remove it? could that be hanging it up? if so, can I run the SAS tools for the LSI HBA that I downloaded manually from the command line to find the disk?

    Any help appreciated. thanks!
     
  4. stevebaynet

    stevebaynet Limp Gawd

    Messages:
    204
    Joined:
    Nov 9, 2011
    Figured it out. since the disk had just started to die, it was still in the pool faulted, but trying to run. this hung up the format command. I let it keep running and eventually the disk was removed and the UI was responsive again.

    I assume we just need to pull the disk, wait a bit for things to refresh, then pop in a new one in the same slot (all slots are full) wait for it to see the disk, then just replace the disk in UI or command line?
     
  5. _Gea

    _Gea 2[H]4U

    Messages:
    3,668
    Joined:
    Dec 5, 2010
    You may check System > Logs if napp-it is basically running or /var/adm/messages at console for reasons

    Other option is check if a activity disk is constantly on when the pool hangs (remove). If the system is hanging completely, remove all datadisks, reboot and insert disk by disk until the system is hanging (bad disk found, remove)

    This is uncritical for a ZFS raid. The pool becomes available when enough disks come back.
     
  6. stevebaynet

    stevebaynet Limp Gawd

    Messages:
    204
    Joined:
    Nov 9, 2011
    I was able to locate the disk, thanks!

    Another question though. I got a failure alert for this disk last night. but just got the same alert tonight. Does it repeat every 24 hours? I dont see anything else faulted so I assume that is it.
     
  7. _Gea

    _Gea 2[H]4U

    Messages:
    3,668
    Joined:
    Dec 5, 2010
    An alert check is done for example every 5 minutes but you propably don't want a mail every 5 minutes so futher alerts with same reason (bad pool) are blocked for 24h.
     
  8. stevebaynet

    stevebaynet Limp Gawd

    Messages:
    204
    Joined:
    Nov 9, 2011
    ok, perfect. next problem (ha! sorry) my google Foo is failing me.

    in the Web UI, I try to replace the disk (which i removed and replaced in same slot with new disk):

    Here is the rpool:

    pool: rpool
    state: DEGRADED
    status: One or more devices has been removed by the administrator.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
    action: Online the device using 'zpool online' or replace the device with
    'zpool replace'.
    scan: resilvered 51.0G in 0h7m with 0 errors on Thu Apr 6 14:10:58 2017
    config:

    NAME STATE READ WRITE CKSUM
    rpool DEGRADED 0 0 0
    mirror-0 DEGRADED 0 0 0
    c1t5000C5002346EEF7d0s0 ONLINE 0 0 0
    c1t5000C500234C8147d0s0 REMOVED 0 0 0

    errors: No known data errors

    When i try to replace the removed disk in napp-it UI, it errors out:

    "cannot replace c1t5000C500234C8147d0 with c5t50000394182AA306d0: no such device in pool"

    I tried from command line to, same:

    # zpool replace -f "rpool" c1t5000C500234C8147d0 c5t50000394182AA306d0
    cannot replace c1t5000C500234C8147d0 with c5t50000394182AA306d0: no such device in pool

    I'm guessing this has to do with the disk being removed? can you point me in the right direction?

    Cheers!
     
  9. stevebaynet

    stevebaynet Limp Gawd

    Messages:
    204
    Joined:
    Nov 9, 2011
    Strange, this is what we had to do instead:

    zpool replace rpool c1t5000C500234C8147d0s0 c5t50000394182AA306d0

    which worked.

    We then (i assume needed since this disk needs to be bootable?):

    installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c5t50000394182AA306d0s0

    Let me know if my logic is sound. I'm no ZFS expert. haha.
     
  10. _Gea

    _Gea 2[H]4U

    Messages:
    3,668
    Joined:
    Dec 5, 2010
    The zpool replace command wants olddisk and newdisk as parameter.
    On Solaris datapools this is always the whole disk or partition so the name ends with d0

    napp-it asumes this as well.
    When like in your case on a mirrorred rpool, you use a slice of a disk (name ends with s0)
    you may indeed need to start the replace command via console manually.

    Probably as you need extra modifications on a bootdisk you may need to remove the disk
    from the bootmirror and add a new one then to rebuild the boot system.