Broadcom Samples World's First 200G Ethernet Controller with PCIe 4.0 and 50G PAM-4

cageymaru

Fully [H]
Joined
Apr 10, 2003
Messages
22,060
Broadcom has begun sampling its third generation E-Series controller called Thor that brings the world's first 200G Ethernet controller with PCIe 4.0 and 50G PAM-4. It is designed for high performance computing, networking and storage applications including machine learning, storage disaggregation, and data analytics. 400G network connectivity will allow for more high-availability features and the 50G PAM-4 SerDes boosts performance and server efficiency. Companies that are adopting and integrating the technology include Lenovo, AMD, NVIDIA, Tencent, and more.

"AMD intends to be an early adopter of PCIe 4.0 to meet the ever-increasing need for efficient, high-performance computing resources," said Raghu Nambiar, vice president of datacenter design engineering at AMD. "Broadcom is a valuable contributor to the technology ecosystem and AMD looks forward to the support for our CPU and GPU processor architectures from the Thor PCIe 4.0 product."

"Broadcom's 200Gb Ethernet controllers continue to improve bandwidth on NVIDIA GPU-accelerated clusters, providing higher performance, and better scalability for HPC and deep learning applications," said Paresh Kharya, group product marketing manager for Accelerated Computing at NVIDIA. "With Broadcom's controllers supporting NVIDIA GPUDirect RDMA technology, data transfers between NVIDIA's data center GPUs can achieve higher bandwidth to address ever more complex and challenging scientific and industrial problems.

 
This is great and all, but can we please first get 10GBaseT in mainstream consumer devices before we go all crazy?
10GBaseT is all but dead, unless you are using Cat6A or better for an install it doesn't actually work at those speeds and internal wiring for it is too easy to interfere with, at best you get 2.5G with it in most installs. For short runs (< 100m) multi-mode OM3 is cheaper and more reliable has a wide market for switches and SFP+ ports, and if you already have SFP+ ports and your run is under 15 feet you can buy the stacking cables for dirt cheap. The only place 10GBaseT had a market was for PoE for high end high density AP's. The spec is also too hot and too power hungry not really making it useful for most home routers and switches.... its faster yeah but it just isn't any good.
 
10GBaseT is all but dead, unless you are using Cat6A or better for an install it doesn't actually work at those speeds and internal wiring for it is too easy to interfere with, at best you get 2.5G with it in most installs. For short runs (< 100m) multi-mode OM3 is cheaper and more reliable has a wide market for switches and SFP+ ports, and if you already have SFP+ ports and your run is under 15 feet you can buy the stacking cables for dirt cheap. The only place 10GBaseT had a market was for PoE for high end high density AP's. The spec is also too hot and too power hungry not really making it useful for most home routers and switches.... its faster yeah but it just isn't any good.


All of these things are true at the enterprise level, but 10GBaseT is the only standard I've seen thus far to replace the aging 1000BaseT standard for home and small business use.

For a long time I had a direct link between my desktop and my NAS at home (a 40ft run) using two brocade BR1020 adapters, OM3 fiber and transducers. It was terrible. Never worked right. and annoying as hell. I was thrilled when some old Intel 10GBaseT server adapters popped up on eBay at an affordable enough price to install at home, and I've been getting great speeds in a direct link between my Desktop and NAS using a 40FT category 7 cable.

I get surprisingly good speeds with this setup. I've actually seen a full 1.2GB/s for sequential file transfers over NFS.

I just wish I could use this standard affordably on my entire network. Until 10GBaseT adapters get down in price to like $40 a pop new, and 24 port managed 10GBaseT layer2 switches get down to about $300
 
I'll even take the 2.5gbit stuff if 10gbit is so bad, plain gigabit isn't fast anymore (well it's pretty good for my internet link.) My raid arrays transfer stuff between each other at 360MB/sec, would be nice to approach that over the network too. The Cat 6 wiring I have may not quite be up to snuff for true 10gbit, but I am sure it would handle 2.5gbit and 5gbit just peachy. I just want a 4 or 8 port switch that doesn't cost an arm and a leg.
 
I'll even take the 2.5gbit stuff if 10gbit is so bad, plain gigabit isn't fast anymore (well it's pretty good for my internet link.) My raid arrays transfer stuff between each other at 360MB/sec, would be nice to approach that over the network too. The Cat 6 wiring I have may not quite be up to snuff for true 10gbit, but I am sure it would handle 2.5gbit and 5gbit just peachy. I just want a 4 or 8 port switch that doesn't cost an arm and a leg.


As I posted above, for me, Copper 10GBaseT hasn't sucked at all. It had been great.


Now granted, my run is only 40ft, but part of me says that of course if you do it wrong it will suck. Get Cat 7 or Cat 7a cables and you should get full performance up to the max length spec (100m/330ft)

If you do something stupid, like try to use old Cat 5e or Cat 6 cables, of course you are going to have problems.


Gigabit Copper Ethernet (1000BaseT):
Designed for Cat 6, but Cat 5e generally works.
Any older cable will give you problems.

10 Gigabit Copper Ethernet (10GBaseT):
Designed for Cat 7, but Cat 6a generally works.
Any older cable will give you problems.

Complaining about 10GBaseT not working on Cat 6 or 5e is a PEBKAC problem.
 
As I posted above, for me, Copper 10GBaseT hasn't sucked at all. It had been great.


Now granted, my run is only 40ft, but part of me says that of course if you do it wrong it will suck. Get Cat 7 or Cat 7a cables and you should get full performance up to the max length spec (100m/330ft)

If you do something stupid, like try to use old Cat 5e or Cat 6 cables, of course you are going to have problems.


Gigabit Copper Ethernet (1000BaseT):
Designed for Cat 6, but Cat 5e generally works.
Any older cable will give you problems.

10 Gigabit Copper Ethernet (10GBaseT):
Designed for Cat 7, but Cat 6a generally works.
Any older cable will give you problems.

Complaining about 10GBaseT not working on Cat 6 or 5e is a PEBKAC problem.
Yeah I am ok with it not being full, I just wish I could get 2.5 or 5gbit switches. Verizon set my 150mbit fios up on cat 3 cable (I will try to find the photo,) I have some cat 5 cable here that has no problem running gigabit speed, so I don't always believe those ratings as iron clad. I'm only looking to go 50' and less anyway here in my basement.
 
I'll even take the 2.5gbit stuff if 10gbit is so bad, plain gigabit isn't fast anymore (well it's pretty good for my internet link.) My raid arrays transfer stuff between each other at 360MB/sec, would be nice to approach that over the network too. The Cat 6 wiring I have may not quite be up to snuff for true 10gbit, but I am sure it would handle 2.5gbit and 5gbit just peachy. I just want a 4 or 8 port switch that doesn't cost an arm and a leg.
Mikrotik 16 port 10Gbit switch is sub $400. On ebay sales w/ 15-20% coupons you can buy them new for just under $300 shipped...
 
Mikrotik 16 port 10Gbit switch is sub $400. On ebay sales w/ 15-20% coupons you can buy them new for just under $300 shipped...
5 port gigabit switches are $10, 8 ports are $15. Even playing wait for coupon games and getting it for $300, that's 4x the ports I want and still kinda pricey. I was really hoping for something in the middle lol.
 
5 port gigabit switches are $10, 8 ports are $15. Even playing wait for coupon games and getting it for $300, that's 4x the ports I want and still kinda pricey. I was really hoping for something in the middle lol.
10G barely made it into the business and enterprise markets, most of us just went past it right to fiber, with out enterprise support I doubt it will trickle down into the cheaper home markets for a while, I suspect it will never really make it and something else entirely will fill in its place instead and 10G will just end up one of those funny technologies that just never caught on.

Its power hungry and hot so it requires active cooling which keeps it out of that price point.
 
Yeah I am ok with it not being full, I just wish I could get 2.5 or 5gbit switches. Verizon set my 150mbit fios up on cat 3 cable (I will try to find the photo,) I have some cat 5 cable here that has no problem running gigabit speed, so I don't always believe those ratings as iron clad. I'm only looking to go 50' and less anyway here in my basement.
Found the photos. The fios installer ran cat 3 for 150mbit service. It worked, but I replaced it right away with my own run anyway. Not really looking to make an ultimate point, just following up because I said I'd find them.
20150927_170457.jpg 20150927_170845.jpg 20150928_174040.jpg
 
Found the photos. The fios installer ran cat 3 for 150mbit service. It worked, but I replaced it right away with my own run anyway. Not really looking to make an ultimate point, just following up because I said I'd find them.
View attachment 103781 View attachment 103782 View attachment 103783
2 of my buildings had gigabit working using interwoven cat3 cables (it was a mess) we rewired with Cat6A but the noise from the florescent lights and the HVAC systems kept interfering with the 10G so we had to give up and go fiber for our switch interconnects. Still have Cat6A there from the switches to the wall jacks which given everything is wireless now basically just runs at 10/100 for the ip phone sets.

The ratings have more to do with the voltage loads over distance than anything else, I have seen plenty of cases where good quality Cat5E outperformed cheap Cat6, but I have also seen cheap Cat5E melt at a kink point under a 10G load.

*Edited: the runs were Cat6A not Cat6
 
Last edited:
Mikrotik 16 port 10Gbit switch is sub $400. On ebay sales w/ 15-20% coupons you can buy them new for just under $300 shipped...


Are they any good? I've never used their switches.

Wasn't there a Mikrotik security hack a while back? Can't remember.
 
2 of my buildings had gigabit working using interwoven cat3 cables (it was a mess) we rewired with Cat6 but the noise from the florescent lights and the HVAC systems kept interfering with the 10G so we had to give up and go fiber for our switch interconnects. Still have Cat6 there from the switches to the wall jacks which given everything is wireless now basically just runs at 10/100 for the ip phone sets.

The ratings have more to do with the voltage loads over distance than anything else, I have seen plenty of cases where good quality Cat5E outperformed cheap Cat6, but I have also seen cheap Cat5E melt at a kink point under a 10G load.


Still, you were trying to get 10G speeds over cat 6, which is below the minimum for the standard, which is 6a.

In a situation like that you might get it to work over short runs, but if you want reliable full speed operation you should be using a cable tested to 6a or above standards.

Again, for me 10GBaseT using Cat7 cabling has worked perfectly at full speeds without any problems.

Again, I'd suspect it was your expectations that it would work on Cat 6 that was the problem, not the standard itself.
 
Are they any good? I've never used their switches.

Wasn't there a Mikrotik security hack a while back? Can't remember.
Their switches are fine ..... I have used a few of their routers in my time but UBNT did everything they did but better for a similar price. Best I can find all the Mikrotiks now only have the SFP+ ports but they claim to make a 10Gbase-T SFP+ module but I can't find anybody who actually sells them. Looking at the Mikrotik forums for them they are filled with people complaining that they don't work at anywhere close to 10G if they can get them to work at all.
 
10GBaseT is all but dead, unless you are using Cat6A or better for an install it doesn't actually work at those speeds and internal wiring for it is too easy to interfere with, at best you get 2.5G with it in most installs. For short runs (< 100m) multi-mode OM3 is cheaper and more reliable has a wide market for switches and SFP+ ports, and if you already have SFP+ ports and your run is under 15 feet you can buy the stacking cables for dirt cheap. The only place 10GBaseT had a market was for PoE for high end high density AP's. The spec is also too hot and too power hungry not really making it useful for most home routers and switches.... its faster yeah but it just isn't any good.

I see 10GBaseT as a better solution for small business, just wish the prices would come down.

I don't need it to the desktop, I need it for server connections, as the 1GB connections are getting to be too slow.
I don't need fiber as 20 foot cat 7 cables would cover everything in the computer room.
The added advantage is that 10GBaseT is backwards compatible with gigabit ethernet, so if a 10GBaseT switch fails and I don't have a spare, I can simple plug the cables into spare gigabit ports.
 
Still, you were trying to get 10G speeds over cat 6, which is below the minimum for the standard, which is 6a.

In a situation like that you might get it to work over short runs, but if you want reliable full speed operation you should be using a cable tested to 6a or above standards.

Again, for me 10GBaseT using Cat7 cabling has worked perfectly at full speeds without any problems.

Again, I'd suspect it was your expectations that it would work on Cat 6 that was the problem, not the standard itself.
That was a typo it is supposed to say 6A, the cable runs are 6A. For the interconnects we could have tried Cat7, but that is more expensive per foot than OM3 with fewer options.
 
Their switches are fine ..... I have used a few of their routers in my time but UBNT did everything they did but better for a similar price. Best I can find all the Mikrotiks now only have the SFP+ ports but they claim to make a 10Gbase-T SFP+ module but I can't find anybody who actually sells them. Looking at the Mikrotik forums for them they are filled with people complaining that they don't work at anywhere close to 10G if they can get them to work at all.

That is too bad.
 
That was a typo it is supposed to say 6A, the cable runs are 6A. For the interconnects we could have tried Cat7, but that is more expensive per foot than OM3 with fewer options.

Ah,

Yeah, as I mentioned before, I tried using a couple of Brocade BR1020 cards with transducers and OM3 cable from The Fiber Store before I went 10GBaseT.

Had so many problems with them that it made me swear off ever touching fiber again.

While I like enterprise hardware, I am very much not an Enterprise user though.

For home and small business I think 10GBaseT is a great solution, if it only came down to affordable prices. There power and long run cable sensitivity is less of an issue.

I have no doubt fiber is king in large scale enterprise implementations.
 
I see 10GBaseT as a better solution for small business, just wish the prices would come down.

I don't need it to the desktop, I need it for server connections, as the 1GB connections are getting to be too slow.
I don't need fiber as 20 foot cat 7 cables would cover everything in the computer room.
The added advantage is that 10GBaseT is backwards compatible with gigabit ethernet, so if a 10GBaseT switch fails and I don't have a spare, I can simple plug the cables into spare gigabit ports.
But just about every switch you can get has 2 or more SFP+ ports on it, most new servers don't even have a 10G option but they do have SFP+ ports as options. You can buy 3m SFP+ patch cables for less than $20, and 1m cables for less than $10, just run those directly from the switch to the server. Yeah you loose backwards compatibility with that but you still have the option of just running a standard cable from the switch to one of the existing 1GB ports on the server should the switch port fail for some reason. You could always just run the cable anyways program the static and disable the port then configure it as a fail over in the event the SFP+ connection fails.
 
Ah,

Yeah, as I mentioned before, I tried using a couple of Brocade BR1020 cards with transducers and OM3 cable from The Fiber Store before I went 10GBaseT.

Had so many problems with them that it made me swear off ever touching fiber again.

While I like enterprise hardware, I am very much not an Enterprise user though.

For home and small business I think 10GBaseT is a great solution, if it only came down to affordable prices. There power and long run cable sensitivity is less of an issue.

I have no doubt fiber is king in large scale enterprise implementations.
Yeah I can't say I have ever met a Brocade I liked, I have had really good experiences with the StarTech PEX20000SFPI, I have a number of them installed in my older servers and they have been solid.
 
I would suggest a used Cisco.

https://www.networktigers.com/products/cisco-ws-c4900m-catalyst-4900m-switch
https://www.networktigers.com/produ...talyst-4900m-8-port-10gbase-t-rj-45-half-card

Contact the vendor to ensure it comes with a Power Supply. I'm using this with all fiber. Have also used some expensive new Netgear 10GB switches. Never had an issue with CAT7 cables.
Those are nice switches, but even used they are close to $1000 for the setup you listed there .... a far cry from the sub $300 most are hoping for. But yeah if I had to go with 10GBase-T for some reason that would be the product line of my choice.
 
This thread has had me start looking at other options lol. I think I'm shifting towards doing some kind of sfp+ run between the server and my pc, skipping the switch as I really only need the speed between these two pcs. Now the trouble is finding what I actually need, in a length that will work for me. Doesn't look like they have keystone adapters for these. But that's getting off topic, I need to do more research and maybe make a thread over in networking.
 
I haven't really ever seen RJ45 Deployed for > 1 Gbit... I wonder what market share is...

Softlayer uses it in their newer deployments. The runs are short though, that's just for rack switch to servers. Seems to work fine, and I think it's easier to get port speed negotiation going (they charge more for 10g than 1g, even though it's the same hardware either way -- i think they sell 100M for cheapskates too; but my boss isn't that cheap, we get 2x1G for most things and 2x10G for the stuff that needs it)
 
This thread has had me start looking at other options lol. I think I'm shifting towards doing some kind of sfp+ run between the server and my pc, skipping the switch as I really only need the speed between these two pcs. Now the trouble is finding what I actually need, in a length that will work for me. Doesn't look like they have keystone adapters for these. But that's getting off topic, I need to do more research and maybe make a thread over in networking.
Leviton will have any cable or wall plate or keystone you can imagine.
 
Leviton will have any cable or wall plate or keystone you can imagine.
Yeah and monoprice actually has one too, so it's not as unheard of as I though, thanks. This weekend I'll get some measurements the the distances I need and make a post over in networking.
 
I wouldn't mind some RJ45 10g for the house, but honestly except for full system backups or massive file transfers, 1g is plenty. Hell, 100mb is fine for web surfing and gaming.

At work we use multi mode fiber OM3 for the 10/16/40g runs to the fiber switch (Brocade) going from server to SAN. The 10g trunk runs are also OM3 fiber but I think that's more of a legacy thing, I've seen new equipment coming in with 10g RJ45 jacks.

It wasn't that long ago (2004 era) that 100mb full duplex was considered optimal for datacenter servers, and only the really high priority stuff got gigabit.

For the longest time I used to collect all the fiber we tossed thinking it was worth something or I'd use it some day. 15 years later a giant hefty bag of it hit the trash.

So I welcome our new super fast ethernet overlords... as long as I can keep using the endless supply of Cat5e and Cat6 I have rescued from dumpster runs.

edit - looks like the 10g rj45 cards are $100... couldn't find a switch under $400 on amazon in the 60 seconds I spent looking.
 
Just a FYI I'm not sure if they have shipped yet but the tallos 2 workstations (power 9 cpu) have pcie 4
 
Back
Top