OmniOS does not want to negotiate 10GbE with Intel X540-T2

S

securitux

n00b
Joined
Jan 31, 2012
Messages
44
Hi all, bit of an issue.

I have built a SM storage server which has dual onboard NIC's, Intel X540-T2's. Mobo is an X9SRH-7TF.

My ESXi host has dual Intel X540-T2's as well.

OmniOS refuses to negotiate to 10Gb, only 1Gb. I am using, for now, a Cat6 cable between them, no switch. I am somewhat confident it's an OmniOS issue because when I connect the dual port nics on the ESXi box together, they come up 10Gb no problem. The Omni ports, always 1Gb. Could always be hardware though :S I am running the latest BIOS...

Any suggestions? The OmniOS box has the newest setup, I am not sure why the issue would be here unless there is something physically wrong with it.

It's not the end of the planet if I need to put on a different OS to test, the box is just in testing phase right now, but I really don't want to have to do that if I can avoid it.
 
Yup. It's detected by the system and according to the OmniOS HCL (they reference the Illumos HCL):

http://illumos.org/hcl/

It's there.

Intel Corporation Ethernet Controller 10-Gigabit X540-AT2

Driver: ixgbe
PCI ID: 8086,1528
Originating manifest: driver-network-ixgbe.mf

It shows ixgbe0 and 1.
 
I'm not familiar with OmniOS. It looks like a linux variant?

I have X540's (in NIC form factor) connected back to back at 10GbE without issues on ESXi 5.5 hosts and RHEL 6.4 hosts. Have you tried updating the drivers?
 
In the category of what the actual f, we have a new winner.

I have a mirror of the OmniOS install so I yanked one of the drives and used it to install OI 151a8. It didn't bring up the link at 10G either.

I then decided to use a Debian live CD and try that, Debian had no problem bringing up the 10G link.

So I decided I'd deal with 1G and contact Omni about the issue. I plugged the OmniOS drive back in, booted, and it came up 10G. What... the... hell. No changes at all to the Omni install.

Maybe it needed a swift kick in the ass from another OS? lol. No idea. I still need to do some tests to make sure it retains the link but for now it seems okay.

-J
 
Can you update the ixgbe driver on OmniOS?

It seems that it's not advertising 10G auto-negotiation correctly. If you have console support, another way to do it would be attempting to force both sides to 10G.
 
dexvx said:
Can you update the ixgbe driver on OmniOS?

It seems that it's not advertising 10G auto-negotiation correctly. If you have console support, another way to do it would be attempting to force both sides to 10G.
Click to expand...

I have no idea if the driver can be updated. OmniOS documentation seems almost non-existent in any useful detail (I guess you use Illumos docs) and community support seems very limited as well (basically, this forum). I couldn't find drivers anywhere for NICs for OmniOS.

I was going to force but I couldn't find the commands to do that for a 10G NIC. From what I read when I did see mention of disabling autoneg, forcing speed and duplex is NOT recommended due to issues such as dropped connections and improper link monitoring. I am unwilling to sacrifice long term stability for temporary functionality.

It's still working now though, at 10G.. I don't understand what happened and I hate it when random crap like this happens with no explanation. It's like when OI decided to drop my iSCSI service a year back. I was never able to recover it, the logs were entirely useless, and searches / questions posted revealed nothing. Eventually I had to blow it away and reinstall, which is not a solution for any OS built in this century.

Anyway, /rant :)
 
securitux said:
I don't understand what happened and I hate it when random crap like this happens with no explanation. It's like when OI decided to drop my iSCSI service a year back. I was never able to recover it, the logs were entirely useless, and searches / questions posted revealed nothing. Eventually I had to blow it away and reinstall, which is not a solution for any OS built in this century.
Click to expand...
This situation is exactly why you have BE for. I suggest you start to use BE, which are ZFS snapshots for the system disk. Before you upgrade your system, create a Boot Environment (i.e. snapshot). If something fails, just reboot into the earlier snapshot and you are back to where you were.

This has saved me many hours of work. I try something crazy, it does not work, I just reboot and delete the latest snapshot, so I know that I should try some other commands. And doing that and breaking something again, I do another rollback to an earlier state, etc.

BE = save games. Before you try some crazy move, just save the game and if you die, reload.

PS. OmniOS is OpenSolarish.
 
Indeed, however when I lost my iSCSI service the OS hadn't had anything done to it in many months. I 'set it and forget it' when it comes to making changes to the NAS. One day it just decided to take a crap with no explanation. It's not like I upgraded or something. If I had, it would have been easy to diagnose. The lack of change or any information on a possible root cause was the problem.

Perhaps it was a hardware fault of some sort, perhaps something got corrupted, but I wouldn't know because the OS was useless for troubleshooting the issue.

I am going to snap the OS drive a lot more on this install just in case something like that happens again though.
 
I create a new BE every time I do something that could be troublesome. I often create BEs anyway - heck, they dont cost anything! No space, no nothing. One guy had someting like 10.000 snapshots (snapshot every 10 sec), and they consumed 1GB in total or so.
 
You must log in or register to reply here.
Back
Top