• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

10gig networking

S

sdadept

Limp Gawd
2FA
Joined
Jul 3, 2004
Messages
433
gigabit ethernet has been around for what seems forever and the sizes of files are getting way beyond what can easily be moved by this protocol. Has anyone managed to do a 10gig ethernet setup in a small business or home? What did you buy? What do you recommend?
 
sdadept said:
gigabit ethernet has been around for what seems forever and the sizes of files are getting way beyond what can easily be moved by this protocol. Has anyone managed to do a 10gig ethernet setup in a small business or home? What did you buy? What do you recommend?
Click to expand...

1gb networking is usually not the bottleneck.

Disk I/O, settings on devices, and even the NICs in play usually are a limit before 1gb.
 
File sizes aren't really getting that high for gigabit... heck, all of our new networking gear at work is still all 10/100. 10 gig ethernet is VERY costly. Just look up the price of a 10g switch. Gigabit will be here for a long time still.
 
sdadept said:
gigabit ethernet has been around for what seems forever and the sizes of files are getting way beyond what can easily be moved by this protocol. Has anyone managed to do a 10gig ethernet setup in a small business or home? What did you buy? What do you recommend?
Click to expand...

First, Gigabit is a standard (IEEE 802.3), not a protocol.

Second, as mentioned, Gigabit is rarely a bottleneck in modern home or small business networks. 125 Megabytes per second is fast... a gigabyte every 8 seconds, 7 gigs a minute, 420 gigs an hour. Typical hard drive transfer speeds are not that much faster, and you'd need 10-Gig NICs as well.

And if it's that much of a problem, you can always just link aggregate Gig ports to make a multi-gigabit link.
 
/usr/home said:
File sizes aren't really getting that high for gigabit... heck, all of our new networking gear at work is still all 10/100. 10 gig ethernet is VERY costly. Just look up the price of a 10g switch. Gigabit will be here for a long time still.
Click to expand...

We just put 100 grand down on 96 ports of 10gb and 12 fexs of cisco gear. Its very expensive indeed but disk I/o is catching up. My ssds at home to my storage arrays can saturate a gig line. Ssds prices are dropping and if the tech keeps improving than ssd based systems will slowly start pushing towards consumer demand for ten gig. Big issue though...internet connections need to catchup before I do see ten gig demand kicking up. Give it a decade :)
 
affordable intel 10gb is already here, so it won't take a decade. the cards are out and board makers like supermicro are including it. network companies are already looking beyond 10gb for the decade...

10gb was expensive trouble up until recently. now that the new intel 10gbase-t 10gbe is out and about, the prices and availability of 10gb stuff will improve quickly. the new chip is small and low power enough to become common. companies selling this stuff intend to roll it out as fast as they can. (there is a lot of money in this upgrade to existing infrastructure. since it just drops in, it is a convenient change to start now and do slowly...)

X540-T2 is what you want to look at
http://ark.intel.com/products/58954/Intel-Ethernet-Converged-Network-Adapter-X540-T2
http://www.intel.com/content/www/us...-network-adapters/ethernet-x540-t2-brief.html

it is the cost of a raid controller. drop it into existing cat6a infrastructure. if switches are too expensive, build your own with two cards. easy!
 
If you are just doing a simple point to point connection between 2 machines, you can team 2 gig channels for a 2GB/s pipe, or you can find a cheap pair of FC 4GB cards and do IPoFC.
 
I have a full 10GBase-T network at home, including a 24 port switch.

The switches still cost a lot unfortunately.
 
For the guys with a 10Gb home network...What are you doing that even comes close to taxing a gigabit network?
 
These days even a low-end laptop with an SSD is bottlenecked by the NIC. I have some servers at home that I use to store several hundred TB. When I want to backup a server or replace a set of drives 1 Gb is too slow.
 
To even come close to achieving the performance available from 10Gig, you have to be running SSDs of some kind or your own SAN array. Even the fastest mechanical hard drives can barely come close to saturating a SATA 1.5GB/s interface.

We installed several blades into our Cisco 6500 series switches several months back at a cost of somewhere around $30k+ per blade.
 
It's not hard to cap gigabit at this point with modern mechanical drives or SSDs. While the cost/benefit balance for moving to 10G is not there for home users yet, that's going to change as disk i/o pushes further and further past the 1Gbps mark.
 
It's going to be a while for good 10Gb to get cheap enough to really be viable for home use. Remember when 1Gb got cheap enough? The components sucked. You'll see the same on 10Gb. I am testing a Synology DS3611xs right now with 4 x 128GB SSD and 8 x 1TB 7200 RPM HD. The DS3611xs has 10Gb and I can pull almost 700MB/sec off those SSDs and close to that off the HDs (sequential reads from a RAID10 set).
 
I just bought a HP 5406zl chassis switch for my core at each site. Just the 10gb 8 port Base-T module cost $3.3k. This however was the cheapest way to get more then 2 10G-baseT ports. The interesting bit is I am only using 10G to link the SAN and ESXi hosts to the LAN. The SAN and ESXi hosts are linked via 40gb infiniband.
 
450 said:
For the guys with a 10Gb home network...What are you doing that even comes close to taxing a gigabit network?
Click to expand...

In my particular case and on my home network, backup to a dual-drive LTO5 changer. Even with theoretical 125MB/sec performance out of GigE, you only generally attain 100 or so due to overhead. 10G gives me the pipe which is enough to handle the ~250-300MB/s which comes into the box with the SAS attached changer. The 3 boxes that send their data to the backup box are capable of 400MB/s+, so I will have the headroom for when I upgrade to LTO6 and need the 300-600MB/s bandwidth for the interleaved streams hitting the backup box.
At work we have a Mellanox FDR switch with gives us ~50Gb/s to a Quad-drive LTO5 array which we should have the LTO6 upgrades soon, and that will have up to 20 servers each capable f pushing 300MB/s going to a D2D2T intermediary.
 
Last edited:
to the trolls posting 'oh you dont need 10g'... I own a small development shop. We have lots of virtual servers on several physical servers that replicate our clients environments as test for the development we do for them. Most of our servers are SSD raid 1+0 with drive read throughputs around 400mb/s. 500gig VM's and multigig data files take hours to move between servers as we reset environments, reconfigure setups, etc... All of this is working fabulously except for moving things between machines.



To those who shared great info, thanks!

Silhouette, what kind of hardware did you use? How much did you spend?

spyder, that sounds pretty close to what I need and almost dead on for our environment. I'll check those out.
 
sdadept said:
to the trolls posting 'oh you dont need 10g'... I own a small development shop. We have lots of virtual servers on several physical servers that replicate our clients environments as test for the development we do for them. Most of our servers are SSD raid 1+0 with drive read throughputs around 400mb/s. 500gig VM's and multigig data files take hours to move between servers as we reset environments, reconfigure setups, etc... All of this is working fabulously except for moving things between machines.



To those who shared great info, thanks!

Silhouette, what kind of hardware did you use? How much did you spend?

spyder, that sounds pretty close to what I need and almost dead on for our environment. I'll check those out.
Click to expand...

You said small business/home. I don't know how many small businesses that you have dealt with, but most are fine on 10/100 for sharing documents, etc. Places that are sharing large files, yeah gigabit may not cut it. I wouldn't lump those in with SOHO. Gigabit is fine for home/SMB.
 
/usr/home said:
You said small business/home. I don't know how many small businesses that you have dealt with, but most are fine on 10/100 for sharing documents, etc. Places that are sharing large files, yeah gigabit may not cut it. I wouldn't lump those in with SOHO. Gigabit is fine for home/SMB.
Click to expand...

Realistically, I would bet that 90% of copper out there is operating at 100mbps. It has only been in the last 5 years (2 really for this matter) that I have seen a push for gigabit in home routers. I'm talking about the every day joe who is going to walk in to Besy Buy and pickup a $60 Netgear router with gigabit, and it is only because the salesman convinced him it was worth it.

For your application, using the cheaper Infinband backbone for the servers and linking it to the core switch via 10gb is probably the best bet. My core is linked to a half dozen switches via 4gb trunks for distribution. We are a heavy use shop, our clusters max out 40gb.
 
Electrofreak said:
First, Gigabit is a standard (IEEE 802.3), not a protocol.
Click to expand...
if you're going to try and be all technical at least use the correct standard. 802.3 is 10BASE5, 802.3z or most commonly 802.3ab is for gigabit. also, 802.3 in general defines the ethernet protocol so ... yeah.
 
sdadept said:
Silhouette, what kind of hardware did you use? How much did you spend?
Click to expand...

Intel 1/2 port NICs, Dell 8024 switch, cat6a cabling. The servers run on a wide range of hardware.

By far the most expensive item was the switch, I paid 3500$ + vat and shipping.
 
As said, 10Gb/s is usually overkill. But I'm finding 1Gb/s can be easily saturated.
However, I'm actually kind of shocked with the performance of "desktop" grade drives I'm building in to some non-critical servers lately.

On fake-RAID (HP SmartArray B110i) in RAID10 with 4x 2TB Seagate Barracuda drives, I'm getting sustained reads over 250MB/s, writes over 220MB/s (testing to/from a SATA SSD). That's better performance on a 3 year old Dell SAS array in RAID10 with 15K drives.
 
Sequential transfer rates are not why you buy the 15K drives. Small, random performance is the reason.
 
madrebel said:
if you're going to try and be all technical at least use the correct standard. 802.3 is 10BASE5, 802.3z or most commonly 802.3ab is for gigabit. also, 802.3 in general defines the ethernet protocol so ... yeah.
Click to expand...

Smarty pants. There are multiple gigabit standards so I just threw the parent standard in there.

Electrofreak said:
First, Gigabit is a standard (IEEE 802.3ab, 802.3z or 802.3ah), not a protocol.
Click to expand...

There, fixed. :rolleyes:
 
Last edited:
mwroobel said:
If you are just doing a simple point to point connection between 2 machines, you can team 2 gig channels for a 2GB/s pipe, or you can find a cheap pair of FC 4GB cards and do IPoFC.
Click to expand...

No you cant, well not really. There is no standard or protocol that supports teaming for 2gb throughput between two individual host as of right now. The only teaming protocol that comes close is 802.3ad and that is not 2gb/s it's actually a balance between two nics that when nic a is saturated nic b will kick in and add to the bandwidth for other host but no one host can get 2gb/s from another host using any protocol out right now save for 10gb/s network bandwidth and associated hardware.

I know what you meant though but it just isn't fully understood. Link ag I mean.

Now my switch in my sig has 2 10gigabit ports so I am ready to go for whenever I am ready and I am not. I cant justify anything right now that I would required 10gbit connection speed in my home and office environments. My switch is total infinite nuclear overkill honestly for what I have connected to it.
 
tangoseal said:
No you cant, well not really. There is no standard or protocol that supports teaming for 2gb throughput between two individual host as of right now. The only teaming protocol that comes close is 802.3ad and that is not 2gb/s it's actually a balance between two nics that when nic a is saturated nic b will kick in and add to the bandwidth for other host but no one host can get 2gb/s from another host using any protocol out right now save for 10gb/s network bandwidth and associated hardware.

I know what you meant though but it just isn't fully understood. Link ag I mean.

Now my switch in my sig has 2 10gigabit ports so I am ready to go for whenever I am ready and I am not. I cant justify anything right now that I would required 10gbit connection speed in my home and office environments. My switch is total infinite nuclear overkill honestly for what I have connected to it.
Click to expand...

Thats what happens when I reply at 2am :) Yeah, you need a switch in the middle. As to how the data is loaded onto the two connections, it really depends on which "teaming" technology is used. For example, Nvidia has their "dualnet" http://www.nvidia.com/docs/CP/45121/dualnet.pdf which load balances the two pipes. The latest beta drivers for the intel 82574 also supports intelligent load balancing over aggregated links, so it isn't just one and then the other. 10GbE gear is coming down in price and there are new PHY chips coming down the pike which should drop per port costs under $100.
 
If you dont need all your ports at 10gig you can get a switch like the hp procurve 2910al (£1500), some Intel X520-DA2 (£500) and Cisco SFP+ Twinaxial 1-10m cables (£75-200).

For 3 ports its costing me about £3500 ($5300).
 
mwroobel said:
Thats what happens when I reply at 2am :) Yeah, you need a switch in the middle. As to how the data is loaded onto the two connections, it really depends on which "teaming" technology is used. For example, Nvidia has their "dualnet" http://www.nvidia.com/docs/CP/45121/dualnet.pdf which load balances the two pipes. The latest beta drivers for the intel 82574 also supports intelligent load balancing over aggregated links, so it isn't just one and then the other. 10GbE gear is coming down in price and there are new PHY chips coming down the pike which should drop per port costs under $100.
Click to expand...

Yeah im really waiting to see these things drop a little more. If I can get 10gig nics for around $100-150 I will load up on a few of em and link my server I am going to be building to my switch and link my desktop sig rig to it also so I can really move some large stuff just for fun haha.
 
NetJunkie said:
$150 for a 10Gb NIC? Eh..no day soon...and if they do they'll be junk.
Click to expand...

I think it will be quite a bit sooner rather than later. 12-18 months at the latest, and it will be everyone not just the cheapo PHY makers. Especially with non-carrier 40GbE and 100GbE starting to sample, as well as QDR and FDR IB to put further pressure on the pricing.
 
I don't care if the NICs are $5, if the switches are still several thousand it doesn't do much good.
 
There will be cheap switches as well. It will start with switches that have 90% GigE / 10% 10GigE ports like Gig switches did, and then ramp down in price as the phy prices decrease and volumes ramp up. If you want a cheap fix right now, you can get some real bargain-basement pricing on 10Gb Infiniband HA's and switches on ebay. A couple of HA's, a switch and some cheap CX4 cables gives you a cheap 10GB IP over IP network.
 
madrebel said:
realtek will make it happen. you'll get maybe 4gbps from them.
Click to expand...

As long as they are priced right, that wouldn't be too bad. If I could spend $50 a card to get 2 10gbe nics that would hit 4gbps that would already speed up transfers between servers substantially. I would just do point to point with a cross-over cable. Once the switches come down in price then I could look into faster solutions and reuse those cards in desktops.
 
bman - see my above post - if you want cheap p2p 10Gb now, go the Infiniband IPoIB now on the cheap.
 
You must log in or register to reply here.
Back
Top