Switch for testing/prototyping 10Gbe rollout?

W

WarlordBB

Limp Gawd
Joined
Jun 10, 2004
Messages
390
We have a 5 node VMWare cluster connected to an EMC NS20 using VMWare over NFS.

Right now, our only bottleneck is network and even that is not limiting. We are considering updating our infrastructure and since networking seems to be the the only real issue, they want me to research that first.

So I thought I would get a server, throw in a 10Gbe NIC, get a cheapo 10Gbe switch, get some cheapo NAS (or build one) that can do 10Gbe and start testing...

Wrong. What a CF 10Gbe still is. At first I just assumed I'd go with 10Gbe over copper. How naive of me. So once I realized that the most "cheapo" 10GBase-T copper switch would run me about $9,000, I shrugged and starting looking at 10Gbe over optics...

Well, long story short, IS there a cheapo (relatively) switch out there for someone wanting to test 10Gbe?? Like $5,000 or less?

Actually, as soon as I know the answer to that I'll be asking about cheapo NAS type solutions so _any_ suggestions you think would help (other than go back to teaming 1Gbe) would be helpful.

TIA

EDIT: BTW, before this goes off-topic into teaming 1Gbe connections being good enough and such, let me say that we are planning for the future here. As I said, we aren't currently limited in any area right now. However, we see that if there was an area we need to focus on, it would be network. So to that end, we'd like to start testing what the future has to offer. We've already tested and are using link aggregation and such at the 1Gbe level. We don't need to research that and we certainly don't need to discuss it here.
 
Last edited:
Google is your friend


HP Procurve Switch 10GBE Copper 6400CL-6XG Mfr P/N J8433ABA.

Enterasys G3G124-24P
 
What are you going to test? Whether you can ping across it or performance? If it's performance than buying cheap test gear is a really bad idea. 10Gb is NOT a commodity thing and gear varies greatly, including NICs.

That NS20 also can't do 10Gb so you'd need to move up to like a VNX. If you do 10Gb to the VMware boxes you need to look at how you plan to traffic shape and/or separate traffic. With vSphere 4.1 vMotion will use up to 8Gb if it's doing 8 vMotions at once...can starve other VMs.

How many VMs do you have on those 5 hosts? How are your Gb connections configured? What is your vSwitch configuration?

10GBase-T is still not mature. The chipsets out now are early and it's a real power hog. Distances are limited and you can't do FCoE over it due to error rate problems. That's why fibre and twinax are still king. Fiber optics cost a lot. Twinax is dirt cheap up to 7m...little longer with active twinax but that costs a good bit more.

EDIT: With only 5 hosts it's probably cheaper to just scale out wider to 6 or 7 hosts.
 
That ProCurve switch is CX4, not 10Gbase-T. 10G-CX4 adapters are available and relatively cheap used. Search ebay. The CX4 cable, however, is stinking expensive. If you go this route be careful about the NICs. There are a lot of them out there that have the 10Gb-CX4 interface but have no on-board network processing to manage offload. At 10Gbe speeds this can lead to being quite burdensome on your host CPU.

Agree with everything NetJunkie said.

If you really want to experiment with 10Gbe then it will be expensive. Plain and simple. Agree that twinax is probably the way to go for now if you must do it. To use it, get NICs that are "DA" (direct attach) or SFP+ and use the SFP-DA twinax cables. DA/SFP+ NICs used range from $400-$700 on ebay (but they are at least dual-port...). An SFP+ switch will still set you back ~$7k used. I don't see any way you get 5 NICs, cables and a switch without dropping almost $10k.

There is hope on the horizon. Both Intel and Broadcom have 25nm 10Gbase-t chips on their roadmaps for release late this year. If they get them out the door you should start to see 10Gbase-T NICs hitting $150 or less new by middle next year and 8 port switches right around $1,000. Still not exactly cheap, but that's down by almost a full order of magnitude from present pricing.
 
Last edited:
You must log in or register to reply here.
Back
Top