FCoE on direct link

[rsq]

Hi Everyone,

I need a quick sanity check on an idea.

Starting position:

I have three servers. One big fat storage box, and 2 compute nodes.

The compute nodes boot of the storage server via fibre channel point to point links.

The compute nodes have 2*gbit ethernet which is too slow for the scientific apps running. The network is the bottleneck.

The compute nodes only have 1 PCIE slot which is occupied by the fibre channel card. (well, they have more, but only one that is not full of GPU)

Idea:
I would buy 3 brocade br1020 adapters and 6 SFP+ gbics. This is about all I can fit in the budget.

Stick one card in each of the servers. Connect the two compute nodes together for the science app comms. Connect both compute nodes to the storage server via the secondary ports.

Would it be possible to export the boot volumes for the compute nodes on FCOE via the Linux SCSI target and have the compute nodes boot from it?

I know that FCOE only works if the switch and all HBA's support the DCB extensions. But what if I eliminate the switch?

Please don't say iSCSI. I need to boot from the storage machine. I could setup the whole DHCP/PXE thing, but I have not a lot of scope for downtime either. (Swapping the cards, and telling the scsi target to export the luns on a different card would be a lot faster.)

Thanks for all insights.

 

[4saken]

I am fairly certain(we have only looked at FCOE years ago and never implemented it since its pmuch a dead tech), that you will require the switch to be the transit for FCOE. So, I'm going to have to say iSCSI. Someone can correct me if that is inaccurate.

 

[Langly]

Check out
http://brasstacksblog.typepad.com/b...tiator-to-an-fcoe-target-part-1-overview.html

Neat little read on differences between parts of the standard and how its evolved.

 

[4saken]

Check out
http://brasstacksblog.typepad.com/b...tiator-to-an-fcoe-target-part-1-overview.html

Neat little read on differences between parts of the standard and how its evolved.

very cool read.

 

[patrickdk]

Does the br1020 support booting from fcoe?

You don't need a switch, but if you put a network port into fcoe target mode, you can't use it for ip normally.

So on your storage server if you want to still use normal networking, you would need something else to handle that, as that would only be able to handle the fcoe.

 

[rsq]

Does the br1020 support booting from fcoe?

You don't need a switch, but if you put a network port into fcoe target mode, you can't use it for ip normally.

So on your storage server if you want to still use normal networking, you would need something else to handle that, as that would only be able to handle the fcoe.

The cards can boot from FCOE no problem. I have all the expected options in the config menu.

From the link of Langly it is clear that the point to point link is not yet implemented.

I guess I will have to do the whole PXE thing. Why the hell don't they support iSCSI boot on all enterprise NICs... PXE is everywhere, but we want iSCSI.