Home-made linux cluster with 10G ethernet

chiu

Limp Gawd
Joined
Sep 14, 2011
Messages
344
Hi,

I am building my own linux cluster but I got a very limited budget.

One node in the cluster will serve as the master node that contains a NFS server.
All other nodes will mount a NFS directory on this master node.
Clearly, the master node will be the bottleneck of the network as all file accesses go to the master node.

I want to upgrade the master node with the new ASUS motherboard that has a 10G ethernet port.
But I have no money to buy 10G ethernet cards for other nodes. So all other nodes will have 1G ethernet ports only.

I think a 24/48port 10G ethernet switch will work well, but they are a bit too expensive for me.
Is there a Ethernet speed that has one 10G ethernet port but many 1G ethernet ports?



TC
 
There are 1 G switches that have 10G uplink ports (2 or 4 10g uplinks).

Have you looked into link aggregation for the master node?
 
As k1pp3r stated, there are a lot of switches that have GbE interfaces on them with 2-4 10G uplinks. A 24/48 port switch with all 10Gb interfaces will definitely cost a lot of cheddar.

What is your budget?

Link Aggregation is a technology used to logically "bind" two or more interfaces to increase throughput and connectivity resiliency from failed links.

https://en.wikipedia.org/wiki/Link_aggregation
 
As k1pp3r stated, there are a lot of switches that have GbE interfaces on them with 2-4 10G uplinks. A 24/48 port switch with all 10Gb interfaces will definitely cost a lot of cheddar.

What is your budget?

My budget is a few hundred bucks but not more than a thousand bucks. Can you suggest a few good switch?

Btw, is an uplink just another ethernet port?
 
My budget is a few hundred bucks but not more than a thousand bucks. Can you suggest a few good switch?

Btw, is an uplink just another ethernet port?

Do you have a 10G NIC for the main syste,? If not that alone will be $300+ for a good one.

You may want to rethink your idea here, not to knock you down, but 10G and Cheap just don't go in the same sentence.
 
Yes - an uplink port is the same as the other interfaces, they are just there as an addition to allow connection to any upstream devices (like another switch).

You also need to figure out what type of medium you are connecting: Fiber? Twin-Ax?

I guess I'm not understanding your use-case for needing 10G. Just because your primary server has 10G available doesn't mean you need to use it.
 
Do you have a 10G NIC for the main syste,? If not that alone will be $300+ for a good one.

You may want to rethink your idea here, not to knock you down, but 10G and Cheap just don't go in the same sentence.

I plan to buy a ASUS X99-E-10G WS motherboard, which has two 10G NICs. Yes, the motherboard is not cheap.
 
Yes - an uplink port is the same as the other interfaces, they are just there as an addition to allow connection to any upstream devices (like another switch).

You also need to figure out what type of medium you are connecting: Fiber? Twin-Ax?

I guess I'm not understanding your use-case for needing 10G. Just because your primary server has 10G available doesn't mean you need to use it.

Can the uplink connect to a computer? No idea about Fiber or Twin-Ax. From the pictures of the ASUS X99-E-10G WS motherboard, the 10G ethernet ports are just like other ethernet ports.

Since the master server is a file system server (NFS), it will be accessed by many computers all the times. Hence, it suffers from a heavy network load. I just want to check whether 10G LAN will help.
 
Do you have a 10G NIC for the main syste,? If not that alone will be $300+ for a good one.

You may want to rethink your idea here, not to knock you down, but 10G and Cheap just don't go in the same sentence.

In his OP he said that he has an Asus motherboard that has built-in 10G interface. I'm assuming something 10Gb Copper SFP+ module on a switch would do this, but again.. more money.

He's easily in the $1000 range on a switch plus the SFP+ module.

chiu - do you need the switch to be L3?
 
Can the uplink connect to a computer? No idea about Fiber or Twin-Ax. From the pictures of the ASUS X99-E-10G WS motherboard, the 10G ethernet ports are just like other ethernet ports.

Since the master server is a file system server (NFS), it will be accessed by many computers all the times. Hence, it suffers from a heavy network load. I just want to check whether 10G LAN will help.

Have you evaluated the speeds of your NFS to verify you will even saturate a 1Gb link?

Yes, uplinks can connect to a computer just the same, but with 10Gb it's not as easy to find a product within the price range.
 
Last edited:
No. How to check?

Uhh, how fast are the disks in your server? At this point I'm wondering how in the world you plan on implementing this if you don't have any idea what this stuff is? I'm trying my best here to not sound flippant, but I don't see how you have scoped your requirements, and I don't see how with your skillset this can be achieved. It sounds like you need some professional assistance with this.
 
Uhh, how fast are the disks in your server? At this point I'm wondering how in the world you plan on implementing this if you don't have any idea what this stuff is? I'm trying my best here to not sound flippant, but I don't see how you have scoped your requirements, and I don't see how with your skillset this can be achieved. It sounds like you need some professional assistance with this.

All disks are SSDs (Samsung 850 EVO). My plan is to hook up 40 desktop computers to one master server.

Yes, my skillset is quite limited. Please help.
 
All disks are SSDs (Samsung 850 EVO). My plan is to hook up 40 desktop computers to one master server.

Yes, my skillset is quite limited. Please help.

Based off this, you don't need 10G.
 
Hi,

I am building my own linux cluster but I got a very limited budget.

One node in the cluster will serve as the master node that contains a NFS server.
All other nodes will mount a NFS directory on this master node.
Clearly, the master node will be the bottleneck of the network as all file accesses go to the master node.

I want to upgrade the master node with the new ASUS motherboard that has a 10G ethernet port.
But I have no money to buy 10G ethernet cards for other nodes. So all other nodes will have 1G ethernet ports only.

I think a 24/48port 10G ethernet switch will work well, but they are a bit too expensive for me.
Is there a Ethernet speed that has one 10G ethernet port but many 1G ethernet ports?



TC

for a 48 port, ubiquiti edgeswitch lite...

24 port, mikrotik
CRS226-24G-2S+RM

also you could probably save yourself some money by not getting 10g onboard your mobo, just ebay a seperate sfp+ nic, that way you can also run fiber sfps all around

or even just DAC cable if you make sure your master is next to your switch
 
Last edited:
On 10G I wouldn't touch a Netgear or any of the lower end switches. I would stick to the people that know how to do fabric's etc. Otherwise you aren't going to get the performance out of the switching gear etc.
 
On 10G I wouldn't touch a Netgear or any of the lower end switches. I would stick to the people that know how to do fabric's etc. Otherwise you aren't going to get the performance out of the switching gear etc.
Generally good advice, but perhaps less important than it used to be in this age of merchant silicon. There are really only a couple of companies making the 10Gbe fabric chips these days - Broadcom, cavium, etc. So the "fabric" for basic 10Gbe is pretty common - even in the low end switches. And they all start out with the same reference software. And at the low-end even the "big boy switches" don't implement additional buffering beyond what the merchant silicon provides. So if your needs are basic layer-2 cut-through switching there is limited value in sticking with the "big boys".

Now - if you are looking for Layer-3 features, routing protocol support, QoS and/or big-buffer switching, etc., then the advice holds true (unless, of course, you really know what you are doing - in which case managing white-box switches can be a better answer).
 
Back
Top