10Gb direct link & routing

omniscence

[H]ard|Gawd
Joined
Jun 27, 2010
Messages
1,311
Recently I got two Intel X540 for a direct connection between my fileserver (Linux) and my desktop (Windows 8). I would like to configure both computers in way that they transparently prefer the fast link over the normal gigabit links to the switch, but only for communication between those two computers. I don't want one of them to act as a router for the other. At the moment I assigned a different subnet the those interfaces and established a network share to the IP of the server. That is not really transparent and does only work for file transfers. Is there a more elegant way to solve this?

One of my ideas was to install a software switch/bridge on the server, add all server interfaces to this bridge and let STP decide which path to use. This may be a too complicated solution, however.
 
Use the routing table (route command) to add a route with a low metric that anything in your LAN subnet goes over the 10Gb link, but your default route stays with your other link. Note that changes to the routing table are only until the OS resets unless you use the permanent flag (can't remember if it's /p or something else offhand but you can do a /? or whatever).
 
The way you are doing it now is the best way to do it. I do not understand what you mean by " is not really transparent." That is the most transparent way as you are now dealing with directly connected devices. You might also want to edit the hosts files of the two 10GE servers and use the 10GE addresses for name resolution.
 
The way you are doing it now is the best way to do it. I do not understand what you mean by " is not really transparent." That is the most transparent way as you are now dealing with directly connected devices. You might also want to edit the hosts files of the two 10GE servers and use the 10GE addresses for name resolution.

This is the easiest option.
 
Ignore the other two posters; my solution is correct (works with hostnames too) and easier.
 
Why do you think he needs to add a route to a directly network? You do not need to manually add a route to directly connected network. You remind me of an Ixia SE we had banned from our site because he didn't understand basic routing.
 
on a windows box you can do a "route add" in a Admin Cmd console and specify an interface to use for that specific route. so if you wanted to keep ALL traffic to/from that server on the 10gig link, thats a possible way to do it. need to use the permanent flag, so it persists thru reboots. otherwise, i'd just use a different subnet and use the fileserver as "DAS" for the win8 box.
 
Because the routing table is the way to set a preferred route, that's what the "metric" value is for.
 
Last edited:
All solutions proposed here should work, although the easiest to implement and troubleshoot is adding the IP address and subnet (not gateway or DNS) to the 10G interface. Make sure the network range is separate from your internet connected network. If you need name resolution, edit your hosts filles.

I may be wrong, but route command requires the next hop router, this situation does not want, have, or need a router.
 
Okay, thinking about it some more, the route command itself may not be necessary after all. However, it could work as well. I'm just trying to think of ways to make this work without having to access the fileserver differently depending on which machine you're on. I don't really like the solution to begin with because 1Gbps + 10Gbe CX4 switches are available for <$200 on Ebay so 10Gbase-T just seems like a bad idea currently due to cost (though at least the cabling is easier, especially through walls/floors). When you're not working with business networks where there are more clients and you need a bunch of security and stuff, I don't think one should be bothering to play around with multiple subnets and all that junk unless studying to become a network admin. KISS.
 
I may be wrong, but route command requires the next hop router, this situation does not want, have, or need a router.

Gateway does not necessarily mean router. When you directly connect two computers with a cable generally you set the gateway to the opposite computer. So you should be able to do the same here. The computer will then look at the route tables to determine that it needs to go out the other link to reach the specified IP. Your computer will know what link to use to reach this machine. You can further this by using the "IfInterface" flag in the route add command. Check the link I posted.
 
Gateway does not necessarily mean router. When you directly connect two computers with a cable generally you set the gateway to the opposite computer. So you should be able to do the same here. The computer will then look at the route tables to determine that it needs to go out the other link to reach the specified IP. Your computer will know what link to use to reach this machine. You can further this by using the "IfInterface" flag in the route add command. Check the link I posted.

I understand what you are saying. The method I proposed does not use a gateway, negating the need for a route.
The route example posted was for a device to access a different network, not just a single machine as requested.
 
Add a route for your local network subnet with the 10GbE connection with a lower metric than anything from your 1Gbps card that might override it, and the OS will use the 10GbE connection for local addresses when possible and fall back to the 1Gbps connection otherwise. Not sure why you keep insisting it can't be done, as it can.

Different subnets and addressing the server by its IP and as a result having to address the server differently from different machines technically works. It's just a half-assed solution which is taking SOME tips from running an enterprise network and trying to fit them into the home network where they don't make sense. Some users here seem to think that home networks should work exactly like corporate networks, but unless you're studying how to set that up, that's just ridiculous.

A Woven LB4 switch (48x1Gbps + 4x10Gbps CX4) with a couple of inexpensive Myricom or Intel NetEffect cards would still be a better setup though. The LB4s are cheap on Ebay; I bought one myself. Only downside to that in general is that the switch is quite loud (I'll be adding some extra heatsinks to mine so that I can replace the fans with quieter ones, but I've been lazy about doing that since it's in the basement anyway). Downside to the OP is obviously he's already bought 10GBase-T stuff (which in a few ways is better, but it's way more expensive which is a pretty major downside for home use, at least at this point in time).
 
Last edited:
I understand what you are saying. The method I proposed does not use a gateway, negating the need for a route.
The route example posted was for a device to access a different network, not just a single machine as requested.

You can use the same route command to add a route to a specific machine also. If you do not add a route with a lower metric the computer may not always use the 10G link. Using the metric flag is a way to guarantee this. I assume the OP would like to force the computer to use the 10G link whenever possible, not just when the OS feels like it.
 
Thanks for the suggestions. I will try to change the routing, I remember trying something like that before, which somehow broke the complete networking and required me to login by IPMI to the server to reset and restore the connection. Maybe I got it wrong. Do I have to disable the ARP filtering of linux?

I considered to buy a switch with a few 10 Gb ports (Cisco SG500X-24 to be more specific, it is not cheap, but I need a larger switch anyway), but those generally have fans and are way to loud for my office.
 
Last edited:
You can use the same route command to add a route to a specific machine also. If you do not add a route with a lower metric the computer may not always use the 10G link. Using the metric flag is a way to guarantee this. I assume the OP would like to force the computer to use the 10G link whenever possible, not just when the OS feels like it.

OK really you need to stop giving advice about routing as you have no idea how it works. This is basic routing it works like this.

Most specific bits to least specific bits lower cost to higher cost when equal bits.

A 32 bit route will ALWAYS be preferred over, for example, a 24 bit route. If the OP uses a route a specific machine this is a 32 bit route. It will always be used unless the OP adds another 32 bit route to the same machine with a lower cost.

Further, directly attached networks will *ALWAYS* prefer the directly attach NICs unless you do something to screw with the routing tables to tell them otherwise. The only potential problem here is that the server names resolve to the gigE links instead of the 10GE links which is why I told the OP to edit the hosts file. This way the OP can use names and the server will always communicate to each via 10GE and to the rest the network via gigE. It is not hard stop trying to make it so.
 
Don't make any changes to your routing table. Configure everything as follows:

Core network - 192.168.0.0/24 (place your standard interfaces on this, with gateway setup to 192.168.0.1).

10-gbps network - 192.168.1.0/24 (configure each 10-gbps interface on this network without any gateway configured).

Create a DNS record or host entry (on your workstation) for your file server as follows:

storage1 - 192.168.0.10 (standard interface)
storage1-xgbe - 192.168.1.10 (10-gbps interface)

When referencing the file share on your computer, use the "storage1-xgbe" address.

This will allow you to avoid playing around with your routing tables, having multiple gateways, or any other shenanigans. This is the best, easiest, and most robust approach.
 
Don't make any changes to your routing table. Configure everything as follows:

Core network - 192.168.0.0/24 (place your standard interfaces on this, with gateway setup to 192.168.0.1).

10-gbps network - 192.168.1.0/24 (configure each 10-gbps interface on this network without any gateway configured).

Create a DNS record or host entry (on your workstation) for your file server as follows:

storage1 - 192.168.0.10 (standard interface)
storage1-xgbe - 192.168.1.10 (10-gbps interface)

When referencing the file share on your computer, use the "storage1-xgbe" address.

This will allow you to avoid playing around with your routing tables, having multiple gateways, or any other shenanigans. This is the best, easiest, and most robust approach.

This. Ignore everything else said in this thread.
 
Redyouch pretty much gave the same instructions I gave with the exception of the leaving out editing the host on the server which should also be done. The OP already has the 10GE nics on different networks.

Given the nature of the question we can presume the OP isn't not running a DNS server.
 
Redyouch pretty much gave the same instructions I gave with the exception of the leaving out editing the host on the server which should also be done. The OP already has the 10GE nics on different networks.

Given the nature of the question we can presume the OP isn't not running a DNS server.

The guy has almost $1000 into 10-gig NICs and a Linux-based file server, and you don't think he has a DNS server?

And my instructions were more succinct... :D
 
I have a pfSense router running a DNS server. The sole reason I was asking for a routing based solution was that I do not want to mess around with different names or IP addresses to select the connection. I now use the hosts file based solution and it seems to do what I wanted. At the moment I cannot think of a circumstance where it will not work as I expect.
 
Back
Top