IPMI nic missing in ESXi 6.5

Joined
Sep 10, 2019
Messages
24
I have just reinstalled ESXi onto my "new" motherboard a SuperMicro H8DGi-F

Essentially out of the 3 physical NICs: the 2 primary are see by vmware, no probs but the IPMI card is just not there. I guess I need to install manually the right vib or something?
Looking at the manual it seems like it runs on a different chipset:
https://www.supermicro.com/Aplus/motherboard/Opteron6000/SR56x0/H8DGi-F.cfm

Here the output from vmware:

Code:
[root@ESXi:~] lspci -v | grep -A1 -i ethernet
0000:02:00.0 Network controller Ethernet controller: Intel Corporation 82576 Gigabit Network Connection [vmnic2]
         Class 0200: 8086:10c9
--
0000:02:00.1 Network controller Ethernet controller: Intel Corporation 82576 Gigabit Network Connection [vmnic3]
         Class 0200: 8086:10c9
[root@ESXi:~] esxcfg-nics -l
Name    PCI          Driver      Link Speed      Duplex MAC Address       MTU    Description
vmnic2  0000:02:00.0 igb         Up   1000Mbps   Full   00:25:90:dc:92:e8 1500   Intel Corporation 82576 Gigabit Network Connection
vmnic3  0000:02:00.1 igb         Up   100Mbps    Full   00:25:90:dc:92:e9 1500   Intel Corporation 82576 Gigabit Network Connection


Does anybody have a good idea on where to get started with this?

Thanks!
 

Blue Fox

[H]F Junkie
Joined
Jun 9, 2004
Messages
11,811
This is expected. The IPMI NIC is only connected to the BMC. You cannot use it as a regular NIC with any OS.
 
Joined
Sep 10, 2019
Messages
24
This is expected. The IPMI NIC is only connected to the BMC. You cannot use it as a regular NIC with any OS.

My old Tyan motherboard had all the ports shared though so i could use any physical port I wanted.
I have tried the IPMI in shared mode (not failover) it words via the other NICs, so no need to have physical connectivity but for a reason I can't explain VMs only can't reach the IPMI ip, LAN devices can though...
SO I'm either looking to have the IPMI port working directly in ESX or a workable way to have the IPMI IP reachable from everywhere via a different port.
 

Blue Fox

[H]F Junkie
Joined
Jun 9, 2004
Messages
11,811
Older motherboards had IPMI sharing the NICs and VLANed things internally. This is no longer the case since most people want to put IPMI on its own network/VLAN for security. It is impossible to get the IPMI NIC working in ESXi or any other OS. It is not physically connected to the rest of the motherboard and only goes to the BMC. The only way to get to get traffic from the other two NICs to the IPMI NIC is going to be via an ethernet cable. Just plug all of them into the same switch and you should be set.

If you aren't aware of how the BMC functions, it is literally a separate computer that provides IPMI functionality. The BMC runs its own OS, has RAM, storage, etc. It just happens to be on the same PCB and therefore can tap into VGA/USB for the main system.
 
Joined
Sep 10, 2019
Messages
24
I hear what you say but I'm not sure about the BMC being completely detached form the other NICs on the board. Check this out from the IPMI settings:

1587800861505.png


Dedisacted is what you described above, The default value is "failover" which essentially means if BMC is not directly connected to an ethernet cable the access will be change to shared with the other NICs so essentially it mean the IPMI IP is reachable from NIC2 and NIC3. "Shared" is forcing this behaviour so no ethernet connectivity/check just share directly.
With this said:

- only 1 ethernet cable into NIC2 and IPMI set to shared my LAN devices and access IPMI no problem

The only issue I'm facing with this set up (regardless of ESXi seen this interface or not) that VMs can't reach it

P.S. I am aware of how BMC works as I've been using this for more than 10 years now, but in my ignorance I always interchange BMC and IPMI are they not the same thing?
 
Last edited:

bigdogchris

Fully [H]
Joined
Feb 19, 2008
Messages
18,216
SuperMicro has the ability to configure a standard Ethernet port as a shared (sideband) IPMI interface which I believe is what you are looking at. It's useful in situations where the dedicated IMPI interface dies (which I have experienced, so I have used this setting before). I've never seen/read any documentation suggesting using the IPMI as a standard Ethernet port. I also have ESXi, Linux, and Windows installed on SuperMicro servers and have never seen the IPMI interface available for use within the OS.
 

Blue Fox

[H]F Junkie
Joined
Jun 9, 2004
Messages
11,811
The BMC has access to all the NICs, however the one dedicated to IPMI is only connected to the BMC. I've run into the same problem you have when sharing NICs actually. Easy fix would be to run 2 cables into a switch.

As for BMC and IPMI, well, a BMC doesn't have to provide full IPMI functionality like KVM/USB passthrough/etc, which is what I think most associate IPMI with. Supermicro provides a BMC board for their JBOD chassis and those just control power, fans, and some sensors for example.
 
Joined
Sep 10, 2019
Messages
24
Thanks guys, an additional cable is the way I have it set it up now. It's just a bit unpractical compared to my old Tyan motherboard as I now need a mini switch next to the server where before I just had a long cable from the router into the server and would have been enough.
Out of all the settings I tried it is just confusing to me that from LAN only IPMI is reachable via LAN2 that's all. If it wasn't working it would be simplier to understand lol!
Correcting my OP and title of this thread my actual problem is: IPMI reachable via LAN but not via the host itself (or VMs) when only one cable is connected on NIC2 and the BMC/IPMI is set to either failover or share. It just don't get into my mind how this is ever possible, it it's shared I would expect a mini bridge to be established between BMC and NIC2 and if that's the case I can only think of some sort of mac filtering/spanning-tree something that would allow LAN->NIC2->BMC but not HOST->NIC2->BMC
 
Last edited:

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,654
My guess: when the host tries to access the BMC via the IPMI interface, it has to first ARP for the MAC address of the IPMI interface. This would go out NIC2. But there is no-one 'out there' to reply to the ARP, since the BMC is on the same NIC. Does that make sense? Try creating a manual ARP entry for the IPMI IP address and see what happens?
 
Last edited:

OFaceSIG

2[H]4U
Joined
Aug 31, 2009
Messages
2,455
Why would you want to use your IPMI port for anything other than OOB management? OOB management is the bee's knees.
 

Dopamin3

Limp Gawd
Joined
Jul 3, 2009
Messages
166
Why would you want to use your IPMI port for anything other than OOB management? OOB management is the bee's knees.

This... The whole point of IPMI port is to be able to run the server headless and easily manage it from any other machine. I don't understand why you need it presented to the OS whether it's a hypervisor or literally any other OS.

If you need more than two physical NIC ports grab a dedicated PCIe card.
 
Joined
Sep 10, 2019
Messages
24
Simple: the IPMI interface provide detailed information of the hardware status. Not all the info is made available/understood by ESX. CPU temperature would be a good practical example: I can only get it via IPMI.

So yes I agree most of the time I connect remotely and my packets arrive from the LAN/NIC2 interface so this issue doesn't exist; however planning for worst case scenario if the VPN is down for whatever reason (and Internet still up) I could TeamViewer onto a VM and access the IPMI from there.

P.S. I don't need more than two ports, opposite I could do using one for everything.
 

Dopamin3

Limp Gawd
Joined
Jul 3, 2009
Messages
166
Simple: the IPMI interface provide detailed information of the hardware status. Not all the info is made available/understood by ESX. CPU temperature would be a good practical example: I can only get it via IPMI.

So yes I agree most of the time I connect remotely and my packets arrive from the LAN/NIC2 interface so this issue doesn't exist; however planning for worst case scenario if the VPN is down for whatever reason (and Internet still up) I could TeamViewer onto a VM and access the IPMI from there.

P.S. I don't need more than two ports, opposite I could do using one for everything.

Use a network monitoring application, such as Zabbix or LibreNMS. You'll have one place where you can see everything on your network and if setup right you can read SNMP data and tons of other data from pretty much any device.

edit: also if you're concerned about your VPN going down just make another VPN on another appliance. Like if you're running OpenVPN on your firewall also setup an alternate OpenVPN (or WireGuard or whatever suits your fancy) on another device and make the necessary firewall/NAT/whatever rules on your firewall so it works how you want it. I don't trust Teamviewer to be secure.
 
Top