10Gbe Questions

N

nitrobass24

[H]ard|DCer of the Month - December 2009
Joined
Apr 7, 2006
Messages
10,466
Im in the process of wiring my new(used) home.

I am running a ton of Cat6 everywhere for light switches, AV distribution, HVAC, keypads, etc.

Well some of the cat6 will be use for the LAN, and I was thinking that since Im in the process of redoing EVERYTHING I might as well grab some 10Gbe stuff for certain things.

My HTPC-DVR, Encoder, and WHS are in this closet and accessed all over simultaneously via the network so it makes the most sense for them to have 10gbe and I would prob add it to my main workstation as well cause I can.

The reason for 10gb on the DVR, WHS, and encoding box is I move a lot of data around on a daily basis between the three, and all the while the WHS is getting hit for streaming.

Im also going to setup a network iSCSI target on the WHS for a project/experiment I want to do. Although that will be on a small dedicated gigabit network.

Heres a diagram I made:
The only thing incorrect is instead of HDMI cables for the TVs Im going to use HDMI over Cat6 using Baluns.
Also the Dell 2716 is what I have now, and I would replace it with a 10gbe.
1zbz1h3.jpg



Questions:
1) I know I dont need 10gb yet, but is there a better option if I want more than just 1gbe?
2) Where do I buy a 10gbe switch, and NICs?
3) Should I consider using ethernet over SAS for the WHS, DVR, Encoder....they are all on the same rack?
4) Am I crazy?
 
Last edited:
i don't know if your aware but 10Gb ethernet equipment is very very expensive still because it still in the early stages of being adopted by even large companies. In my opinion its just not worth it for any home user at this point in time. One thing you could do however is get like a dual or even quad port intel NIC and providing your network switch supports it create a team which will allow greater bandwidth. But being a relatively new person to the wonderful world of networking thats all i will say, let the real pro's tell you all the facts.
 
Yea I do gigabit teaming on my WHS right now and it works ok.

But for my other equipemt I would have to buy upgraded NIC's and I was thinking that if I have to buy stuff I might as well go all out....but again I have no idea the cost involved so that may be prohibative.
 
You do realize that unless your HTPC is loaded with 15K SAS drives or a few SSD's, you're going to have a dickens of a time maxing out that 10Gbe lan? If you really need the bandwidth boost over regular GigE, you could investigate using Fiber Channel, 4gb FC HBA's are a heck of a lot cheaper than 10GbE and you can run IP over Fiber Channel too.

Not only that, but you've got a few devices, so you'd need a 10GbE switch, which is most definitely going to cost you. If you really must have 10Gb speeds in the house, check out Infiniband as well, 10Gb and generally more affordable.
 
Man that's going to be costly. The cards are in the range of 600-1000 each and the switch is in the range of 4k upwards.
 
cymon said:
You do realize that unless your HTPC is loaded with 15K SAS drives or a few SSD's, you're going to have a dickens of a time maxing out that 10Gbe lan? If you really need the bandwidth boost over regular GigE, you could investigate using Fiber Channel, 4gb FC HBA's are a heck of a lot cheaper than 10GbE and you can run IP over Fiber Channel too.

Not only that, but you've got a few devices, so you'd need a 10GbE switch, which is most definitely going to cost you. If you really must have 10Gb speeds in the house, check out Infiniband as well, 10Gb and generally more affordable.
Click to expand...

You mean something like this?
Im going to use the 1 TB version, and then all recordings upon completion will be automatically put into the Encoders Queue, and then stored on the WHS.

valve1138 said:
Why do you think you need 10Gbe? GigE is fine for all of this.
Click to expand...

GigE is going to be the limiting factor for some of this.
I wanna go fast:cool:
Ockie said:
Man that's going to be costly. The cards are in the range of 600-1000 each and the switch is in the range of 4k upwards.
Click to expand...

Damn 4k for a switch thats a little more than i was thinking, the Nics yea $600 ok, but 4k kinda blows any sort of budget.

If I wanna go faster than 2gbE should I just get Quad port Intel Nics and bigger switch or should i grab some fiber stuff.
 
10gig is not designed for the home user or budget for that matter. It is meant for enterprise businesses who push a lot of data with VMs, huge NAS storage arrays, and router/switch uplinks. When 40/100gig hits the market hard in a year or two, 10gig will naturally come down in price just like gig.

Most of the products out there can't do it effectively -- their bus sucks. One of the decent non-chassis switches that can handle it is the Cisco Nexus 5k, which is $13000 on CDW. Not to mention how expensive new 10gig SFPs are -- $1000-$2000/ea.

So yeah, 4x gig NIC teaming is the way to go if you get one of those OCZ SSD drives. Unless you have a completely disposable income :D
 
Ethernet teaming is an option, however, you should check out Infiniband, if you're not afraid of buying used equipment then it's a decent option for getting a full 10-20Gb/s network.
 
Wrong, it has the module bays, not four 10Gbe ports, and its not that great an idea to buy that and then say 4-5 years later try to find the modules, they get to be hard to find and/or expensive unless its cisco, then its easy to find and expensive.
NETGEAR GSM7328S 10/100/1000Mbps + 10 Gigabit Gigabit L3 Managed Stackable Switch with 4 10-Gigabit Ethernet Module Bays 24 x RJ-45, 4 x SFP, 4 x 10 Gigabit Ethernet/24G Stacking Module Bays, 1 x RS-232 334 KB embedded memory per port Buffe - Retail
Click to expand...
 
Wouldn't it be more cost efficient to have two networks (physically or vlan) that seperates streaming from regular file transfers? That alone should increase bandwidth.
 
Keiichi said:
Wouldn't it be more cost efficient to have two networks (physically or vlan) that seperates streaming from regular file transfers? That alone should increase bandwidth.
Click to expand...

How will Vlans increase my bandwidth?
 
nitrobass24 said:
How will Vlans increase my bandwidth?
Click to expand...


vlans would breakup broadcast domains and separate traffic across the network fabric...in theory it should increase performance, but on networks nowadays(10Mb+ switched) you'd probably never notice any difference for a home.

I'm seriously trying to figure out why you need to move ~1GB of data a second, short of massive SANs, backbones/uplinks and iSCSI targets i cant see a "need" for that...if you're talking about lag or slow response times, this sounds more like the hardware/servers arent quite up to your standards for what you're asking it all to do....only fix for that is to offload tasks to different servers or increase performance on the current hardware to try and compensate.

your best bet is going to be grab a Cisco or a HP procurve, get some intel quad NICs and go to town...

my .02....
 
cymon said:
Ethernet teaming is an option, however, you should check out Infiniband, if you're not afraid of buying used equipment then it's a decent option for getting a full 10-20Gb/s network.
Click to expand...

The problem with teaming is that it only works with multiple "conversations". One server talking to one client will only use one NIC. To use two you need two clients to something else.

Infiniband is a completely different animal. If you balk at the 10Gb pricing you won't do IB, plus driver support is highly questionable and protocols to do simple things like file sharing are not nearly as defined as IP/Ethernet.

OP: Do some tests. See if you actually hitting a limit with Gb. If it is one box to one other box then options are limited. If it's multiple to one or multiple to multiple you can just team/port channel/bind NICs.
 
From what the OP mentioned, he's got dedicated boxes for encoding, playback, and storage. Therefore, the storage box at least has a potential for multiple concurrent access, which is where an etherchannel would be helpful.
 
I only would really like the increased speed between the DVR, Encoder, and WHS.

As far as NEEDing it...i prolly dont need it, but I would like it for those three machines, especially for the WHS because it handles all the Data.
 
cymon said:
From what the OP mentioned, he's got dedicated boxes for encoding, playback, and storage. Therefore, the storage box at least has a potential for multiple concurrent access, which is where an etherchannel would be helpful.
Click to expand...

Well I have dedicated Recording, Encoding, and Storage in the Central Closet.
There are also 4 computers throughout the house that access the central storage sometimes all simultaneously.

So yea the WHS is gonna need something.
 
Do testing and find out what your real bottle necks are.

My bet is that you are going to be hard pressed to bring a single gigabit link to its knees unless you have some very nice storage hardware.

If you do have very nice storage hardware then as mentioned, look into gigabit etherchannel. You could potentially have your main storage server running an Intel 4 interface PCI-e gigabit card and then run them into a good switch with a strong back plane. Then for your other servers get 2 interface PCI-e gigabit cards and etherchannel them into the same switch.

Use some synthetic memory based benchmarking tools to test everything out to make sure you're getting satisfactory speeds.
 
cymon said:
Ethernet teaming is an option, however, you should check out Infiniband, if you're not afraid of buying used equipment then it's a decent option for getting a full 10-20Gb/s network.
Click to expand...

The increase in throughput from gigabit ethernet to 10gb IP over IB isn't all that much though, and cabling/etc is quite a bit more involved. Sustained I hit 180MB/s, which is about double the speed of what I do on gigabit. The drivers and software stacks are a whole other animal as well. Unless he's using Vista, ESX, Linux or a Windows server OS, don't look for too much support.
 
Asgorath said:
Do testing and find out what your real bottle necks are.

My bet is that you are going to be hard pressed to bring a single gigabit link to its knees unless you have some very nice storage hardware.

If you do have very nice storage hardware then as mentioned, look into gigabit etherchannel. You could potentially have your main storage server running an Intel 4 interface PCI-e gigabit card and then run them into a good switch with a strong back plane. Then for your other servers get 2 interface PCI-e gigabit cards and etherchannel them into the same switch.

Use some synthetic memory based benchmarking tools to test everything out to make sure you're getting satisfactory speeds.
Click to expand...

My Mac Pro with two drives in RAID0 can do 200MB/sec in sequential reads. That's almost double what a single Gb link will provide. So I have no doubt someone moving large media files can do it on a pretty simple system. The question is if the increased cost and complexity are worth it. For 4 other systems I'd just buy a quad card, assign a different IP to each card, point a different system to each card. That's all you'd gain with teaming anyway and you don't need anything other than a dumb switch to do it. You don't want to manually balance 20 systems to one, but 4 or 5 to 1? Easy. The problem is still that you only get one NIC's bandwidth per host.
 
pissboy said:
The increase in throughput from gigabit ethernet to 10gb IP over IB isn't all that much though, and cabling/etc is quite a bit more involved. Sustained I hit 180MB/s, which is about double the speed of what I do on gigabit. The drivers and software stacks are a whole other animal as well. Unless he's using Vista, ESX, Linux or a Windows server OS, don't look for too much support.
Click to expand...

What did you use to test IP over IB and see it wasn't much faster? Was surely faster for me.
 
This is very off topic, but if you don't mind me asking, what sorts of cool tricks can you do with network cable on the cheap? I'm re-wiring my new home, and as soon as the stuff comes in the mail I'm going to do a few runs of cat6 for the LAN, but if there's something else cool that I can do without spending an arm and a leg I'd love to try it out. You said you were running cable for lights, AV, HVAC, keypads, etc... can you elaborate?
 
panfist said:
This is very off topic, but if you don't mind me asking, what sorts of cool tricks can you do with network cable on the cheap? I'm re-wiring my new home, and as soon as the stuff comes in the mail I'm going to do a few runs of cat6 for the LAN, but if there's something else cool that I can do without spending an arm and a leg I'd love to try it out. You said you were running cable for lights, AV, HVAC, keypads, etc... can you elaborate?
Click to expand...

Home automation. Lights control. Music through out the whole house. Control it from the keypads. HVAC is most likely AC and has it controlled by some computer (i'm guessing)
 
NetJunkie said:
What did you use to test IP over IB and see it wasn't much faster? Was surely faster for me.
Click to expand...

I only need a high speed link for moving/backing up files from one machine to another. With the gigabit link I was in the ~90-105MB/s range sustained. Without playing around with the settings IPoIB was maxing out at around 120MB/s. When I used fastcopy and some other apps to copy/move the files I was able to hit 160-180MB/s sustained. Sure I doubled the bandwidth I could use, but the line speed went up 10 fold. Unfortunately IPoIB doesn't use RDMA, so it is very limited in bandwidth compared to native IB and the protocols that support RDMA. I have a case of 20gb HCAs coming in next week, as well as a pair of 10Gbase-CX4 nics, so hopefully those might come closer to maxing out my arrays. :)
 
nitrobass24 said:
So yea the WHS is gonna need something.
Click to expand...
Are you kidding? Can your WHS box even sustain over 125MB/sec for multiple clients? 10GbE is so beyond overkill for this it's not even funny. If you really feel like it, take your server box and team a NIC to something like a Dell PowerConnect 2824. Team 4 1GbE ports together to serve to the rest of your clients and you'll be fine, and even that'll be overkill. Seriously. I'm sure you can find better things to spend that kind of money on.
 
ikari303 said:
Seriously. I'm sure you can find better things to spend that kind of money on.
Click to expand...

Ive already bought all the porn I can find.
Now I just need to be able to watch it REALLY fast.:D

No but seriously I want a single link that is faster than 2gbps.
I thought 10gbe would be the next step, but I am open to suggestions.
 
Get good Gigabit cards for all your machines and you will be impressed.

Most use barely any of gigabit speeds as it is hard to get it optimized and then you are usually limited by your devices.

I bet you are not maxing gigabit at all right now.
 
If you really need more than a gig, etherchannel or link aggregation is the "reasonable" way to go. It will give you a maximum of 8gbps by using 8x 1000Mbit NIC's. Make sure your NIC drivers support it, and grab a gig switch that supports it. Load balancing will occur across all links in the port channel group so you won't need to worry about the 1gb/host problem.
 
is etherchannel just another name for Link aggregation, teaming, bonding, etc.?
 
They accomplish the same end-goal of adding bandwidth, but they function differently.

Etherchannel handles load balancing more gracefully than traditional link aggregation through the use use of a different packet queuing algorithm. With Etherchannel, all your nics act as one logical NIC with a single MAC address and thus can have a communications session with a given host span multiple links.

I believe the traditional type runs above L2, and does not have a common MAC address. It's method of keeping sessions in order is to only use one link per session. Someone please correct me if I'm wrong :)

Either one will probably work just fine realistically, but if you really need to send more than 1 gig to a certain host by itself, go with Etherchannel and a gigabit switch that supports it.
 
Nitro,

Etherchannel is just cisco's term. It is the same thing as Link aggregation, teaming, bonding, etc. As Lightworker said it does use a different load balancing algorithm.

Correct me if i'm wrong, but I think its a total of 16Gbps total bandwidth (8GbpsTX/8GbpsRX) using 8x 1Gb links. At full speed, not accounting for overhead, you would have a theoretical maximum of 1000MB/s of throughput in each direction. A little overkill for a home network =)
 
Shockey said:
Home automation. Lights control. Music through out the whole house. Control it from the keypads. HVAC is most likely AC and has it controlled by some computer (i'm guessing)
Click to expand...

I am seeking out more specific information. So, for example, let's say I make a run of network cable to my thermostat...do I need to bust out some circuit CAD software to hook up a NIC and controller package to the thermostat? or are there off the shelf products that don't have a 5000x profit margin?

Any books out there for someone with a basic knowledge in circuit design to implement DIY home automation?

I was looking through ShockValue's home build thread, and searched some of the components he mentioned like the Nuvo control panel, and that seemed WAAY too expensive for what it did.

I'm not really that interested in music throughout the house, because the layout of my house is such that you can hear music from the home theater pretty much throughout the house, except for our office (which is where all the music will be served from) and the master bedroom (only music needed is bom chikka BOW WOW).
 
nitrobass24 said:
Ive already bought all the porn I can find.
Now I just need to be able to watch it REALLY fast.:D

No but seriously I want a single link that is faster than 2gbps.
I thought 10gbe would be the next step, but I am open to suggestions.
Click to expand...

You could look into 4 or 8 Gbps Fibre Channel.
 
panfist said:
I am seeking out more specific information. So, for example, let's say I make a run of network cable to my thermostat...do I need to bust out some circuit CAD software to hook up a NIC and controller package to the thermostat? or are there off the shelf products that don't have a 5000x profit margin?

Any books out there for someone with a basic knowledge in circuit design to implement DIY home automation?

I was looking through ShockValue's home build thread, and searched some of the components he mentioned like the Nuvo control panel, and that seemed WAAY too expensive for what it did.

I'm not really that interested in music throughout the house, because the layout of my house is such that you can hear music from the home theater pretty much throughout the house, except for our office (which is where all the music will be served from) and the master bedroom (only music needed is bom chikka BOW WOW).
Click to expand...

This is off-topic, so I'll keep it brief. Home automation is still fairly expensive these days. Prices have come down a lot as computers are much cheaper now days and you generally don't need a lot of computing power for home automation. Check out Proliphix Thermostats for a network enabled HVAC controller.

Nitro - I wouldn't discourage you from 10GBe, because 10GBe in a home setting would definitely be [H]ard. That being said, I'd stick with link aggregation for the time being as it's the best bang for your buck. Grab a couple of those Intel Quad Nics and go to town.
 
panfist said:
I am seeking out more specific information. So, for example, let's say I make a run of network cable to my thermostat...do I need to bust out some circuit CAD software to hook up a NIC and controller package to the thermostat? or are there off the shelf products that don't have a 5000x profit margin?

Any books out there for someone with a basic knowledge in circuit design to implement DIY home automation?

I was looking through ShockValue's home build thread, and searched some of the components he mentioned like the Nuvo control panel, and that seemed WAAY too expensive for what it did.

I'm not really that interested in music throughout the house, because the layout of my house is such that you can hear music from the home theater pretty much throughout the house, except for our office (which is where all the music will be served from) and the master bedroom (only music needed is bom chikka BOW WOW).
Click to expand...


Well I havent decided which home automation product I am going to use yet.
For the thermostat, you would have to buy a new thermostat.
Reason being is that most home automation uses the X10 protocol so it needs to be compatible.
While this is still expensive, it has become much cheaper and easier to do recently.
There is software that you can run on a PC, I havce been looking at http://www.charmedquark.com/ for my setup.

There is lots of good info at smarthome.com and home-automation.org.
Also check out AVS forums home automation forum. There are a lot of very knowledgable peeps there.
 
nitrobass24 said:
Well I havent decided which home automation product I am going to use yet.
For the thermostat, you would have to buy a new thermostat.
Reason being is that most home automation uses the X10 protocol so it needs to be compatible.
While this is still expensive, it has become much cheaper and easier to do recently.
There is software that you can run on a PC, I havce been looking at http://www.charmedquark.com/ for my setup.

There is lots of good info at smarthome.com and home-automation.org.
Also check out AVS forums home automation forum. There are a lot of very knowledgable peeps there.
Click to expand...

X10 is long gone by the wayside. It's Z-wave, Insteon, and others now. Much more advanced than X10. I use Homeseer as the brains of my system which includes Insteon switches and a Proliphix thermostat. Smarthome is a great source.
 
You must log in or register to reply here.
Back
Top