OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,989
As ZFS stores all Raid informations on disks, this is save. You only need to

- export pool, power down
- move HBA
- power up, import pool
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,989
rpool (operating system) is normally not on a disk provided by a HBA but a local ESXi datastore ex Sata bootdisk for ESXi.
 

Captainquark

Weaksauce
Joined
Dec 14, 2011
Messages
103
Tried that, but it seems it did something to passthrough settings. Can't turn on the NAS VM, it tells me "Failed to start the virtual machine. Device 101:0.0 is not a passthrough device." Any idea what went wrong or how to solve this?
Thanks!

Edit: checked the config of the VM, there's a PCI Device, but the dropdown behind is empty and cannot be selected. I could only remove it. Maybe once removed, I can add the device again fresh?

Another edit: so obviously, moving the card into another slot gave it another path. I had to go in the host configuration and enable the controller for passthrough again, then reboot the host. Then I had to remove the PCI-passthrough device from the NAS VM and boot it. Once it was up, I shut it down again and added the "new" passthrough device. Only then, it could boot again. Now I imported the pool again and everything works as a charm.
Cheers!
 
Last edited:

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,989
so obviously, moving the card into another slot gave it another path. I had to go in the host configuration and enable the controller for passthrough again, then reboot the host. Then I had to remove the PCI-passthrough device from the NAS VM and boot it. Once it was up, I shut it down again and added the "new" passthrough device. Only then, it could boot again. Now I imported the pool again and everything works as a charm.
Cheers!

That is correct
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,687
So I'm in the process of setting up the HA configuration. I've elected to go with mgmt == LAN. One question: for security, quite some time ago, I put all of the IPMI interfaces on my servers on a special VLAN that is only accessible from my win10 workstation. I note that the two HA servers (bare metal) and the control VM (ESXi) need access to the IPMI addresses. Is the way to fix this to add vlan access on the two physical hosts, as well as the control VM? Re-reading your PDF, it's not clear if the HA members need IPMI access for stonith, as the PDF says "VM reset command for head 1 via ipmi or ESXi cli (for tests enter echo 1)".
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,989
A failover from an active head1 to a standby head2 happens under control of the cluster controlserver/VM. This means that the controlserver initiates a fast remote shutdown of head1 followed by a pool import on head2, a failover of the HA ip and optionally a restore of services like iSCSI or www or a user/group restore from the former active head.

For this you normally do not need an additional kill command (stonith/ shot the other node in the head) of head1. But if head1 hangs for whatever reason and head2 imports the pool it can become corrupted. This is why a second independent kill mechanism of a former active head is implemented. If the whole cluster is virtualised, this can be a VM reset via SSH to ESXi. With a barebone server you can initiate a hard reset via ipmi.

In both cases only the controlserver needs access to ESXi management or the ipmi interface ex via an additional nic or vnic there. If you do not need this additional security, you can skip/fake this step (" echo 1" simulates a successfull stonith ). The heads do not need ESXi or ipmi access.


btw
The multihost ZFS property is already in Illumos. This may be a n additional option to stonith to protect a pool.

but read
https://illumos.topicbox.com/groups...ture-multiple-import-protection-for-ha-setups

more
https://openzfs.org/w/images/d/d9/05-MMP-openzfs-2017.4.pdf
 
Last edited:

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,687
A failover from an active head1 to a standby head2 happens under control of the cluster controlserver/VM. This means that the controlserver initiates a fast remote shutdown of head1 followed by a pool import on head2, a failover of the HA ip and optionally a restore of services like iSCSI or www or a user/group restore from the former active head.

For this you normally do not need an additional kill command (stonith/ shot the other node in the head) of head1. But if head1 hangs for whatever reason and head2 imports the pool it can become corrupted. This is why a second independent kill mechanism of a former active head is implemented. If the whole cluster is virtualised, this can be a VM reset via SSH to ESXi. With a barebone server you can initiate a hard reset via ipmi.

In both cases only the controlserver needs access to ESXi management or the ipmi interface ex via an additional nic or vnic there. If you do not need this additional security, you can skip/fake this step (" echo 1" simulates a successfull stonith ). The heads do not need ESXi or ipmi access.


btw
The multihost ZFS property is already in Illumos. This may be a n additional option to stonith to protect a pool.

but read
https://illumos.topicbox.com/groups...ture-multiple-import-protection-for-ha-setups

I can confirm multihost present.
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,687
A failover from an active head1 to a standby head2 happens under control of the cluster controlserver/VM. This means that the controlserver initiates a fast remote shutdown of head1 followed by a pool import on head2, a failover of the HA ip and optionally a restore of services like iSCSI or www or a user/group restore from the former active head.

For this you normally do not need an additional kill command (stonith/ shot the other node in the head) of head1. But if head1 hangs for whatever reason and head2 imports the pool it can become corrupted. This is why a second independent kill mechanism of a former active head is implemented. If the whole cluster is virtualised, this can be a VM reset via SSH to ESXi. With a barebone server you can initiate a hard reset via ipmi.

In both cases only the controlserver needs access to ESXi management or the ipmi interface ex via an additional nic or vnic there. If you do not need this additional security, you can skip/fake this step (" echo 1" simulates a successfull stonith ). The heads do not need ESXi or ipmi access.


btw
The multihost ZFS property is already in Illumos. This may be a n additional option to stonith to protect a pool.

but read
https://illumos.topicbox.com/groups...ture-multiple-import-protection-for-ha-setups

Thanks, makes my job much simpler :)
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,687
Hopefully one last question: what does 'Net Link head 2 name of the nic link ex vmxnet3s and add :3 ex vmxnet3s0:3 (HA ip will use this link)' mean? Is this intended to be a dedicated heartbeat link between the two hosts?
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,989
Normally you add an ip to a network link ex vmxnet3s0 ex the LAN adress. Under this ip you can always access the server. Additionally you can add or remove one or more ip adresses to the same network link temporarily and name the link with an additional :n (numbers must not be ongoing, you can use :8 without the need of lower numbers set)

You see these numbers also when you call ifconfig -a

In a cluster environment you access services like nfs or smb not over the "normal" link but under a moveable HA ip that is either provided by head1 or head2. The controlserver must know in advance under which link number it can set or access the ip ex vmxnet3s:3 on a per link base (The number is hardcoded in the management software, you should not use for other purposes)
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,687
I'm trying to compare this with the pacemaker/corosync HA ZFS I was using before. So in my case, 192.168.3.0/24 is the 10gb storage network. Host #1 is 192.168.3.44 and host #2 is 192.168.3.45. I intend to use 192.168.3.40 as the HA IP that vsphere will connect to. If I am understanding you correctly, in the CC configuration: HA IP is 192.168.3.40, Net Link Head 1 (primary storage) would be 'aggr0:3' (I have 2 10gb in a bond), whereas Net Link Head2 (old sandy bridge standby storage server), it would be i40e0:3, since the standby server does not have a bonded 10gb link. Does this sound correct?
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,989
OmniOS Security Update r36m, r34am, r30cm - Security / Features/ Bugfixes

Support for secure RPC for the Netlogon protocol between OmniOS systems and Microsoft Active Directory servers is added to all OmniOS versions under support. This is required to fully mitigate CVE-2020-1472, and will shortly be enforced by Windows domain controllers.

If you use Windows Active Directory you should at least evaluate.

https://omniosce.org/releasenotes.html
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,687
Well, I'm not sure what I am doing wrong. I'm trying to follow the instructions in the Z-RAID pdf, but no matter what I do, after I enter all the settings in the CC appliance, it fails totally. Here is the screenshot enclosed. BTW, I think I was not clear - I have 10.0.0.0/24 as LAN and 192.168.3.0/24 as 10gb storage. I've tried setting appliance LAN in either subnet, and neither works. It was also not clear if I'm to set the two appliances in cluster-failover mode before setting up CC appliance, but I've done both ways with no difference.

p.s. I appreciate your patience here...
 

Attachments

  • image0003.jpg
    image0003.jpg
    73.9 KB · Views: 0

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,687
Very strange. I tried to do 'pkg update'. It works on 1 of the 3 installs (omnios1) but not omnios2 or omnios-cc. I get this:

WARNING: Errors were encountered when attempting to retrieve package
catalog information. Packages added to the affected publisher repositories since
the last retrieval may not be available.

Errors were encountered when attempting to contact repository for publisher 'extra.omnios'.

Unable to contact valid package repository: https://pkg.omniosce.org/r151036/extra/
Encountered the following error(s):
Framework error: code: E_SSL_CACERT (60) reason: SSL certificate problem: certificate has expired
URL: 'https://pkg.omniosce.org/r151036/extra'

Errors were encountered when attempting to contact 2 of 2 repositories for publisher 'omnios'.

Unable to contact valid package repository: https://pkg.omniosce.org/r151036/core/
Encountered the following error(s):
Framework error: code: E_SSL_CACERT (60) reason: SSL certificate problem: certificate has expired
URL: 'https://pkg.omniosce.org/r151036/core'

Unable to contact valid package repository: https://pkg.omniosce.org/r151036/core/
Encountered the following error(s):
Framework error: code: E_SSL_CACERT (60) reason: SSL certificate problem: certificate has expired
URL: 'https://pkg.omniosce.org/r151036/core'

on both failing hosts. I verified that /etc/ssl/pkg/OmniOSce_CA.pem. I'm certainly no SSL expert, and didn't do anything that I am aware of. I can also confirm their cert is NOT expired (if I visit those URLs from a browser, all is well.) I can only conclude something is broken in the omnios installs? I have no idea why this happened (I certainly wasn't messing with pkg information, or deleting things, and 2 of my 3 installs are borked?) I suppose I can reinstall, but without some idea what the heck happened, doesn't give me a warm feeling...
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,989
A "certificate expired" happens when the OmniOS cerificate is indeed expired (unlikely) or when the system date is wrong (too old or in future). On ESXi VM date is in sync with ESXi server date.
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,687
Well, their cert is certainly not expired. I never thought of the system date. I will check, thanks!
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,687
Okay, looking good. I've got ipmi working using ipmitools. I tested the failover time by running a linux guest with the dd command using 'status=progress', from /dev/sda to /dev/null, starting a manual failover, and starting a timer when disk I/O stopped, and stopping the timer when it resumed. Seems to be on the order of 45 seconds or so, which sounds ok.
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,687
Hopefully last step: enabled and turned on the failover service (auto mode). Tested by using ipmi manually to power off the master. It all seems to work. The one thing I'm wondering, is in the Appliance Cluster page, ipmi is showing a red indicator?
 

Attachments

  • image0005.jpg
    image0005.jpg
    71.2 KB · Views: 0

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,989
The check executes a console command and tests the result so you need for ex an ipmi status info,
For tests or without stonith this can be a simple "echo 1" and test for a return value of 1.
 
Top