![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
Living Weapon
Limp Gawd
- Joined
- Dec 21, 2007
- Messages
- 448
Infiniband is fun to play with - I've used IB and 10GE at work. For those PCI-x IB cards, be aware that you'll need a server-class motherboard to really take advantage of the bandwidth: 64bit x 133MHz = 1064 megabytes/s, which is half the full-duplex bandwidth of a 10 gig IB port. The card will work in a, a 32bit PCI slot, but a typical desktop board is probably driving that slot at 66MHz at the most. So in a regular motherboard you'll only get 32bit x 66Mhz = 266 megabytes/s. "Modern" IB cards are all PCI-e x8, though if you have a desktop board with an extra PCI-e x16 slot maybe an IB card will work in it - then again, the BIOS on a desktop MB probably expects a GPU in that slot, if anything at all, so YMMV.
All that said, comparing IB vs. 10 gig ethernet - you'll probably get more throughput at lower CPU overhead from an IB setup vs. the same hosts running current generation 10GE cards. However, the diference is small, and completely negligible on multi-core systems. In fact, given the maturity of TCP over ethernet, you may have fewer issues with 10GE at this time.
In terms of driver support, check out http://www.openfabrics.org/ for free IB drivers. You'll need to install the Service Manager (SM) on at least one machine in your IB network in order for any of them to talk. You can directly connect 2 machines back to back with IB, but to get a 3rd machine in the mix you'll actually need to set up additional "subnets" for each PC to PC link - and additional instances of the SM. Of course if you have an IB switch you only need one SM per network. You may want to configure a secondary SM on another machine so your IB network still works when the primary SM host is powered off. The OFED stack from openfabrics includes all the software you need for the SM, a DHCP server, and TCP/IP over IB.
Bridging or routing between IB and ethernet is not very mature as of yet. There are some expensive managed IB switches that do it, but what most folks with IB are doing is setting up a private IB network within a cluster of machines. If you have a few machines, putting them all on IB is the way to go. You can use one of those machines to route to the internet, thereby providing internet access to the other machines.
All that said, comparing IB vs. 10 gig ethernet - you'll probably get more throughput at lower CPU overhead from an IB setup vs. the same hosts running current generation 10GE cards. However, the diference is small, and completely negligible on multi-core systems. In fact, given the maturity of TCP over ethernet, you may have fewer issues with 10GE at this time.
In terms of driver support, check out http://www.openfabrics.org/ for free IB drivers. You'll need to install the Service Manager (SM) on at least one machine in your IB network in order for any of them to talk. You can directly connect 2 machines back to back with IB, but to get a 3rd machine in the mix you'll actually need to set up additional "subnets" for each PC to PC link - and additional instances of the SM. Of course if you have an IB switch you only need one SM per network. You may want to configure a secondary SM on another machine so your IB network still works when the primary SM host is powered off. The OFED stack from openfabrics includes all the software you need for the SM, a DHCP server, and TCP/IP over IB.
Bridging or routing between IB and ethernet is not very mature as of yet. There are some expensive managed IB switches that do it, but what most folks with IB are doing is setting up a private IB network within a cluster of machines. If you have a few machines, putting them all on IB is the way to go. You can use one of those machines to route to the internet, thereby providing internet access to the other machines.
Looks like you added some more GP drives 
