Interesting experience with boot from san

D

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,715
I wanted to play around with this, so I took a pro/1000 server adapter and flashed the iscsi boot firmware onto it. Had some fun getting the vsphere5 installer to work with this, since all the google fu I found told me I needed to have the option rom load but the install cd had to come before the nic, which was fine. The blogs I found showed broadcom nics where you could tell the nic 'do not boot from this target!', and after installing, you change that to 'boot from this target', but there is no obvious equivalent for the intel nics :( By trial and error, I got it to work by changing the iscsi boot port to secondary (disabled mode causes the option rom to not even load, it seems.) So, the installer finds the iscsi LUN (which was on a freenas NAS I had set up), and starts writing to it. It gets to 90% and fails, saying 'Expecting 2 bootbanks, found 0.' A reboot ended up booting that install anyway, although some things like the password were not set. I googled for this, but found nothing very useful - a couple of blog items involving cisco ucs blades, netapp appliances, etc... Some folks were reporting that this worked in 4.1, so I'm curious what this is all about?
 
Yeah, I figured :) I'm more wondering if someone can explain what went wrong? Also, curious as to what needs to be supported - it's an iSCSI target. It seems to be able to actually boot from it, just something went wiggy during the install (for the life of me, I can't figure what could cause a bizarro bootbanks message...
 
No, the NIC isn't an IBFT supported device. AFAIK, the only ones that are are the BNX2 cards (and the only ones that have the needed driver bits). And, of course, true hardware cards.

Install didn't go right, or the process modifying something with the bootbanks didn't go right - no idea, as boot from san is silly (imho), and honestly no one really uses the IBFT boot much. Just slap a USB stick in there, or use autodeploy :p
 
Yeah, I know. I have the USB stick now. I just wanted to try it out, and was surprised at the odd failure. I had the NIC lying around anyway. Thanks :)
 
I was actually thinking of that, but at least one other person here http://cknic.wordpress.com/tag/cisco/ had an experience which seemed to indicate it was a networking glitch (e.g. even though providing static IP, at some point the installer tried to do dhcp - they had no dhcp server on that subnet so it failed.) My current firewall/dhcp is a VM, so it obviously was not running. I'm going to try this again with a temp dhcp server running and see what happens... If this fails, I may try to score one of those broadcom adapters for cheap...
 
Last edited:
Well, I think there is an esxi5 bug here. The guy on that website I linked was right. When it got to 90%, I hit alt-f12 to look at the logs. Despite that the NIC iSCSI settings are configured statically for 10.0.0.8 (and the initial connection to the LUN uses that IP), late in the game, esxi5 installer seems to switch to DHCP - since I had no DHCP server at that point, any subsequent iSCSI writes will not work. It seems to be the installer specifically, since once I got the install to work okay (by setting up a temporary dhcp server), a boot from the intel NIC worked just fine. I know you don't think BFS is worth spit, but honestly, this sounds like a bug.
 
Last edited:
Unsupported iBFT NIC ;) Bug won't go anywhere. Doesn't happen on the BNX2i cards for some reason, so...?
 
Can you point me to a link for what *is* supported? The HCL is not very useful, as that NIC *is* supported as a NIC. Various docs I've found on the vmware site only refer to the NIC needing to support IBFT, which this one does...
 
When you search the HCL look under NICs, not iSCSI. Then click on the notes of the adapter you're curious about and it'll say if IBFT is supported.
 
Ah, okay, thanks. So there's no way to search for any ibft supporting NICs aside from trial and error?
 
Yeah, it's kinda dumb like that. The PDF version may be easier to search - I haven't checked. As far as I know, the only straight network card that we support for IBFT is the BNX2 cards.
 
I scored a bcm5708 on ebay for $25. Will retest with that. FWIW, I think you may be making an unwarranted assumption about it not happening with bnx2i cards - few people are likely to be doing any kind of install like this on a network with no dhcp server, in which case they wouldn't see this. I will post my results with the new card.
 
Yeah, but he's not going to go through the motions on an unsupported NIC. So get a supported one and we'll see.
 
Understood. I work in support myself, so I know how it goes. That's exactly what I am going to do...
 
There is a reason support is so limited on this. I have no one doing boot from iSCSI right now. We may be doing a couple soon now that Cisco UCS supports it, but that's on a very specific known platform. Outside of that...zero people we deal with (probably 200 customers) do it.
 
NetJunkie said:
There is a reason support is so limited on this. I have no one doing boot from iSCSI right now. We may be doing a couple soon now that Cisco UCS supports it, but that's on a very specific known platform. Outside of that...zero people we deal with (probably 200 customers) do it.
Click to expand...

I wasn't even aware it was possible with the VMware software iSCSI adapter until I was studying for the VCP5. Sounds neat that it's an option but I wouldn't recommend it.
 
I hear you guys. On the other hand, this is a classic chicken&egg thing. By definition, no one is using a feature when it is newly introduced. While it *worked* in 4.1, it doesn't seem to have been well-known (and I read more than one blog posting where people had issues with it...)
 
It's not chicken & egg. Features are driven by customer demand. No one is/was demanding it. Most people have a mis-placed hatred of BFS. It's often a tough sell even with UCS until I show the benefits you get from it and how easy it is to do with UCS. People tried it 10 years ago, had a bad experience, and now won't let it go so it's not in demand.
 
danswartz said:
I scored a bcm5708 on ebay for $25. Will retest with that. FWIW, I think you may be making an unwarranted assumption about it not happening with bnx2i cards - few people are likely to be doing any kind of install like this on a network with no dhcp server, in which case they wouldn't see this. I will post my results with the new card.
Click to expand...

Most people aren't doing the install at all. Boot from iSCSI doesn't see much demand, to be honest. That being said, I've seen it in 4.1 with static IPs work fine, and did it once myself in training on an integrated BNX2 card. ~shrug~

danswartz said:
I hear you guys. On the other hand, this is a classic chicken&egg thing. By definition, no one is using a feature when it is newly introduced. While it *worked* in 4.1, it doesn't seem to have been well-known (and I read more than one blog posting where people had issues with it...)
Click to expand...

Lots of features that Engineering folks think are useful, and no one else does :p

danswartz said:
What bad experience? Is it different now?
Click to expand...

Reliability, performance issues, looped dependencies...

Is it different now? Eh... Eh... meh.
 
You must log in or register to reply here.
Back
Top