FreeBSD/Freenas issue/question.

Discussion in 'Linux/BSD/Free Systems' started by /dev/null, Mar 30, 2019.

  1. /dev/null

    /dev/null [H]ardForum Junkie

    Messages:
    13,826
    Joined:
    Mar 31, 2001
    Hey guys,

    I think this is a better forum than storage for my issue, as I think this is more of a FreeBSD question than straight FreeNAS.


    Story so far:

    I have a freenas 11.2 setup with a DL380G6 and an MSA60. This setup has been working very well for a very long time.
    Yesterday my MSA60 went offline completely & so did my entire pool (2 vdevs of raidz2, all disks exist in the MSA60). I tried to reboot my FreeNAS box via CLI but it hung so I hard powered off.

    The boot drives are a ZFS mirror that are LOCAL storage inside the DL380 G6, not in the MSA60.

    Now when it tries to boot, the DL380 G6 goes into a boot loop. It no longer boots as if it can't find a boot sector on either of the ssds in the freenas-boot pool. As a troubleshooting step, I booted a FreeNAS livecd and dropped to a shell. I noticed that my ssds gpt were "corrupted" (from gpart show) and a "gpart recover <disk>" seemed to fix this.

    I was then able to "zpool import -f freenas-boot" and scrub it. Everything looks good. A ZFS list shows what looks like OS files.

    Howvever, the system still doesn't boot.

    So I guess I have 2 issues:

    1) Why did the MSA60 go offline?
    2) Why was the system then unbootable even though "freenas-boot" pool is on internal, locally attached storage?

    I think I need to fix #2 & then #1.

    I have a few ideas to recover

    Idea #1
    0) Boot FreeNAS livecd & mount the filesystem(s) on SSD(s) manually to recover config data.
    q1) Where is the config data? If I can mount the filesystems I'm fine copying it somewhere temporarily.

    1) Unplug all disks
    2) Install 2 new SSDS and install FreeNAS
    3) Re-import pool
    4) Restore config data from step #0
    5) Re-attach MSA60.

    Idea #2
    0) Figure out how to restore boot sector to existing SSDS and try to boot.
    1) Once booting, shut down & then troubleshoot the MSA60

    Anyone run into an issue like this? What path would you take to fix this?
     
  2. /dev/null

    /dev/null [H]ardForum Junkie

    Messages:
    13,826
    Joined:
    Mar 31, 2001
    So after A little more googling it appears I want the freenas-v1.db in /data.

    However, which is the correct zfs filesystem to pull from? freenas-boot/ROOT/11.2-RELEASE-U1 ?

    upload_2019-3-30_17-20-3.png
     
  3. B00nie

    B00nie [H]ardness Supreme

    Messages:
    7,823
    Joined:
    Nov 1, 2012
    I don't know this system at all but I would probably try to boot using the grub.
     
  4. /dev/null

    /dev/null [H]ardForum Junkie

    Messages:
    13,826
    Joined:
    Mar 31, 2001
    I'm thinking the safest option at this point is probably to:
    * remove 2 existing OS ssds with data
    * disconnect MSA60 enclosure
    * put in two new ssds
    * install freenas (11.2-RELEASE-U1)
    * try to load backup (from 11.2-RELEASE-U1)

    At worst, it doesn't work.

    If it does work & I see all my shares but am missing my vdevs, I'll try to re-attach the MSA60 & see what happens.
     
  5. jardows

    jardows [H]ard|Gawd

    Messages:
    1,459
    Joined:
    Jun 10, 2015
    I haven't messed with ZFS or FreeNAS, just base FreeBSD with UFS. I am interested to see what solution you find.
     
  6. /dev/null

    /dev/null [H]ardForum Junkie

    Messages:
    13,826
    Joined:
    Mar 31, 2001
    So my SSDs came in & I had some time to toy with this.

    Took out my old 60G ssds and put in my new 240G ssds. Installed 11.2-RELEASE-U1, loaded backup. All my shares/config look to be there! Still can't see my MSA60 array.

    Shut everything down and reseated fan module as well as cable on BOTH ends (array & raid card which I'm using only single disks as "vdevs") and it made a click on the host/card side.

    Rebooted again & at freebsd boot screen it saw a bunch of disks.

    It's now "importing" the zpool....not sure how long this is going to take, but I imagine it's similar to having a very large filesystem that needs an fsck on mount..

    It may take a while (36T raw...)
     
  7. Spartacus09

    Spartacus09 Gawd

    Messages:
    783
    Joined:
    Apr 21, 2018
    Glad you got it sorted out, this is my nightmare and has kept me from doing a Freenas/ZFS array as I'm not super strong in Linux.
    unRAID looks really nice on the rebuild/recovery/loss aspect and has been tempting me recently.
     
  8. /dev/null

    /dev/null [H]ardForum Junkie

    Messages:
    13,826
    Joined:
    Mar 31, 2001
    Freenas is FreeBSD so quite a bit different than linux.
    With that said I find linux easier....
     
  9. jardows

    jardows [H]ard|Gawd

    Messages:
    1,459
    Joined:
    Jun 10, 2015
    Most Linux distributions will auto-config things for you, so are easier to setup than FreeBSD base. But when something breaks in FreeBSD, since you did all the configuration yourself, it's usually easier to find what went wrong and fix it. Setting up a FreeBSD box is akin to setting up a Gentoo or Slackware Linux system. But in the FreeBSD world, you have the same thing with FreeNAS and pfSense - lots of pre-configured stuff. Lot's easier to setup your NAS or firewall, but when things break, harder to troubleshoot. But I really don't want to setup a firewall from scratch, so I use pfSense, despite the drawbacks.

    Glad you got it sorted out. Sounds more like it was a hardware issue than configuration issue?
     
  10. /dev/null

    /dev/null [H]ardForum Junkie

    Messages:
    13,826
    Joined:
    Mar 31, 2001
    Something went sideways.

    The array dissapeared completely, but then why would it cause the box not to boot (unless the boot sector - BIOS mode..) got written to the front of the array?

    I mean I suppose that is a possibility that freenas put it there, don't know why though. When I checked the bios of the storage controller it was set to boot ONLY from ssd, or fail.

    In any case, when I reinstalled it I made sure the array was completely physically disconnected so freenas would (hopefully) put the boot sector on the SSDS itself. I think there was more than one problem personally..