Broadcom Samples World's First 200G Ethernet Controller with PCIe 4.0 and 50G PAM-4

Discussion in '[H]ard|OCP Front Page News' started by cageymaru, Sep 13, 2018.

  1. cageymaru

    cageymaru [H]ard|News

    Messages:
    18,630
    Joined:
    Apr 10, 2003
    Broadcom has begun sampling its third generation E-Series controller called Thor that brings the world's first 200G Ethernet controller with PCIe 4.0 and 50G PAM-4. It is designed for high performance computing, networking and storage applications including machine learning, storage disaggregation, and data analytics. 400G network connectivity will allow for more high-availability features and the 50G PAM-4 SerDes boosts performance and server efficiency. Companies that are adopting and integrating the technology include Lenovo, AMD, NVIDIA, Tencent, and more.

    "AMD intends to be an early adopter of PCIe 4.0 to meet the ever-increasing need for efficient, high-performance computing resources," said Raghu Nambiar, vice president of datacenter design engineering at AMD. "Broadcom is a valuable contributor to the technology ecosystem and AMD looks forward to the support for our CPU and GPU processor architectures from the Thor PCIe 4.0 product."

    "Broadcom's 200Gb Ethernet controllers continue to improve bandwidth on NVIDIA GPU-accelerated clusters, providing higher performance, and better scalability for HPC and deep learning applications," said Paresh Kharya, group product marketing manager for Accelerated Computing at NVIDIA. "With Broadcom's controllers supporting NVIDIA GPUDirect RDMA technology, data transfers between NVIDIA's data center GPUs can achieve higher bandwidth to address ever more complex and challenging scientific and industrial problems.

     
    Pieter3dnow likes this.
  2. HaloSVT

    HaloSVT n00bie

    Messages:
    53
    Joined:
    Sep 20, 2011
    Its sure taking them a hot minute to get this version out
     
  3. /dev/null

    /dev/null [H]ardForum Junkie

    Messages:
    13,921
    Joined:
    Mar 31, 2001
    Armenius and cageymaru like this.
  4. DNMock

    DNMock Limp Gawd

    Messages:
    189
    Joined:
    Apr 16, 2015
    Sweet, now all they gotta do is actually release PCIE 4.0
     
    Armenius likes this.
  5. Zarathustra[H]

    Zarathustra[H] Official Forum Curmudgeon

    Messages:
    25,279
    Joined:
    Oct 29, 2000
    This is great and all, but can we please first get 10GBaseT in mainstream consumer devices before we go all crazy?
     
  6. Zarathustra[H]

    Zarathustra[H] Official Forum Curmudgeon

    Messages:
    25,279
    Joined:
    Oct 29, 2000
    I could be wrong, but I think th ePCIe 4 standard went final in June of last year.

    There is even a preliminary PCIe 5.0 standard since June of last year.

    I guess what we are waiting for is for Intel and AMD to start integrating the standard into their CPU packages.
     
  7. DNMock

    DNMock Limp Gawd

    Messages:
    189
    Joined:
    Apr 16, 2015
    Ohh, I never saw that. I knew it was close but didn't know the standard had been set already. Thanks!
     
  8. Lakados

    Lakados Gawd

    Messages:
    654
    Joined:
    Feb 3, 2014
    10GBaseT is all but dead, unless you are using Cat6A or better for an install it doesn't actually work at those speeds and internal wiring for it is too easy to interfere with, at best you get 2.5G with it in most installs. For short runs (< 100m) multi-mode OM3 is cheaper and more reliable has a wide market for switches and SFP+ ports, and if you already have SFP+ ports and your run is under 15 feet you can buy the stacking cables for dirt cheap. The only place 10GBaseT had a market was for PoE for high end high density AP's. The spec is also too hot and too power hungry not really making it useful for most home routers and switches.... its faster yeah but it just isn't any good.
     
    shpankey, DocNo and /dev/null like this.
  9. Lakados

    Lakados Gawd

    Messages:
    654
    Joined:
    Feb 3, 2014
  10. Zarathustra[H]

    Zarathustra[H] Official Forum Curmudgeon

    Messages:
    25,279
    Joined:
    Oct 29, 2000

    All of these things are true at the enterprise level, but 10GBaseT is the only standard I've seen thus far to replace the aging 1000BaseT standard for home and small business use.

    For a long time I had a direct link between my desktop and my NAS at home (a 40ft run) using two brocade BR1020 adapters, OM3 fiber and transducers. It was terrible. Never worked right. and annoying as hell. I was thrilled when some old Intel 10GBaseT server adapters popped up on eBay at an affordable enough price to install at home, and I've been getting great speeds in a direct link between my Desktop and NAS using a 40FT category 7 cable.

    I get surprisingly good speeds with this setup. I've actually seen a full 1.2GB/s for sequential file transfers over NFS.

    I just wish I could use this standard affordably on my entire network. Until 10GBaseT adapters get down in price to like $40 a pop new, and 24 port managed 10GBaseT layer2 switches get down to about $300
     
    Armenius likes this.
  11. arnemetis

    arnemetis 2[H]4U

    Messages:
    2,095
    Joined:
    Aug 2, 2004
    I'll even take the 2.5gbit stuff if 10gbit is so bad, plain gigabit isn't fast anymore (well it's pretty good for my internet link.) My raid arrays transfer stuff between each other at 360MB/sec, would be nice to approach that over the network too. The Cat 6 wiring I have may not quite be up to snuff for true 10gbit, but I am sure it would handle 2.5gbit and 5gbit just peachy. I just want a 4 or 8 port switch that doesn't cost an arm and a leg.
     
  12. Zarathustra[H]

    Zarathustra[H] Official Forum Curmudgeon

    Messages:
    25,279
    Joined:
    Oct 29, 2000

    As I posted above, for me, Copper 10GBaseT hasn't sucked at all. It had been great.


    Now granted, my run is only 40ft, but part of me says that of course if you do it wrong it will suck. Get Cat 7 or Cat 7a cables and you should get full performance up to the max length spec (100m/330ft)

    If you do something stupid, like try to use old Cat 5e or Cat 6 cables, of course you are going to have problems.


    Gigabit Copper Ethernet (1000BaseT):
    Designed for Cat 6, but Cat 5e generally works.
    Any older cable will give you problems.

    10 Gigabit Copper Ethernet (10GBaseT):
    Designed for Cat 7, but Cat 6a generally works.
    Any older cable will give you problems.

    Complaining about 10GBaseT not working on Cat 6 or 5e is a PEBKAC problem.
     
  13. /dev/null

    /dev/null [H]ardForum Junkie

    Messages:
    13,921
    Joined:
    Mar 31, 2001
    I haven't really ever seen RJ45 Deployed for > 1 Gbit... I wonder what market share is...
     
    Armenius likes this.
  14. arnemetis

    arnemetis 2[H]4U

    Messages:
    2,095
    Joined:
    Aug 2, 2004
    Yeah I am ok with it not being full, I just wish I could get 2.5 or 5gbit switches. Verizon set my 150mbit fios up on cat 3 cable (I will try to find the photo,) I have some cat 5 cable here that has no problem running gigabit speed, so I don't always believe those ratings as iron clad. I'm only looking to go 50' and less anyway here in my basement.
     
  15. /dev/null

    /dev/null [H]ardForum Junkie

    Messages:
    13,921
    Joined:
    Mar 31, 2001
    Mikrotik 16 port 10Gbit switch is sub $400. On ebay sales w/ 15-20% coupons you can buy them new for just under $300 shipped...
     
    Zarathustra[H] likes this.
  16. drescherjm

    drescherjm [H]ardForum Junkie

    Messages:
    13,925
    Joined:
    Nov 19, 2008


    I thought this was not going to happen until Zen4.
     
  17. arnemetis

    arnemetis 2[H]4U

    Messages:
    2,095
    Joined:
    Aug 2, 2004
    5 port gigabit switches are $10, 8 ports are $15. Even playing wait for coupon games and getting it for $300, that's 4x the ports I want and still kinda pricey. I was really hoping for something in the middle lol.
     
  18. Lakados

    Lakados Gawd

    Messages:
    654
    Joined:
    Feb 3, 2014
    Virtually non existent outside specialized implementations such as for high density AP's
     
  19. Lakados

    Lakados Gawd

    Messages:
    654
    Joined:
    Feb 3, 2014
    10G barely made it into the business and enterprise markets, most of us just went past it right to fiber, with out enterprise support I doubt it will trickle down into the cheaper home markets for a while, I suspect it will never really make it and something else entirely will fill in its place instead and 10G will just end up one of those funny technologies that just never caught on.

    Its power hungry and hot so it requires active cooling which keeps it out of that price point.
     
  20. arnemetis

    arnemetis 2[H]4U

    Messages:
    2,095
    Joined:
    Aug 2, 2004
    Found the photos. The fios installer ran cat 3 for 150mbit service. It worked, but I replaced it right away with my own run anyway. Not really looking to make an ultimate point, just following up because I said I'd find them.
    20150927_170457.jpg 20150927_170845.jpg 20150928_174040.jpg
     
    Armenius, Ranulfo and Lakados like this.
  21. Lakados

    Lakados Gawd

    Messages:
    654
    Joined:
    Feb 3, 2014
    2 of my buildings had gigabit working using interwoven cat3 cables (it was a mess) we rewired with Cat6A but the noise from the florescent lights and the HVAC systems kept interfering with the 10G so we had to give up and go fiber for our switch interconnects. Still have Cat6A there from the switches to the wall jacks which given everything is wireless now basically just runs at 10/100 for the ip phone sets.

    The ratings have more to do with the voltage loads over distance than anything else, I have seen plenty of cases where good quality Cat5E outperformed cheap Cat6, but I have also seen cheap Cat5E melt at a kink point under a 10G load.

    *Edited: the runs were Cat6A not Cat6
     
    Last edited: Sep 13, 2018
  22. Zarathustra[H]

    Zarathustra[H] Official Forum Curmudgeon

    Messages:
    25,279
    Joined:
    Oct 29, 2000

    Are they any good? I've never used their switches.

    Wasn't there a Mikrotik security hack a while back? Can't remember.
     
  23. Zarathustra[H]

    Zarathustra[H] Official Forum Curmudgeon

    Messages:
    25,279
    Joined:
    Oct 29, 2000

    Still, you were trying to get 10G speeds over cat 6, which is below the minimum for the standard, which is 6a.

    In a situation like that you might get it to work over short runs, but if you want reliable full speed operation you should be using a cable tested to 6a or above standards.

    Again, for me 10GBaseT using Cat7 cabling has worked perfectly at full speeds without any problems.

    Again, I'd suspect it was your expectations that it would work on Cat 6 that was the problem, not the standard itself.
     
    shpankey likes this.
  24. Lakados

    Lakados Gawd

    Messages:
    654
    Joined:
    Feb 3, 2014
    Their switches are fine ..... I have used a few of their routers in my time but UBNT did everything they did but better for a similar price. Best I can find all the Mikrotiks now only have the SFP+ ports but they claim to make a 10Gbase-T SFP+ module but I can't find anybody who actually sells them. Looking at the Mikrotik forums for them they are filled with people complaining that they don't work at anywhere close to 10G if they can get them to work at all.
     
  25. nutzo

    nutzo [H]ardness Supreme

    Messages:
    6,966
    Joined:
    Feb 15, 2004
    I see 10GBaseT as a better solution for small business, just wish the prices would come down.

    I don't need it to the desktop, I need it for server connections, as the 1GB connections are getting to be too slow.
    I don't need fiber as 20 foot cat 7 cables would cover everything in the computer room.
    The added advantage is that 10GBaseT is backwards compatible with gigabit ethernet, so if a 10GBaseT switch fails and I don't have a spare, I can simple plug the cables into spare gigabit ports.
     
  26. Lakados

    Lakados Gawd

    Messages:
    654
    Joined:
    Feb 3, 2014
    That was a typo it is supposed to say 6A, the cable runs are 6A. For the interconnects we could have tried Cat7, but that is more expensive per foot than OM3 with fewer options.
     
  27. Zarathustra[H]

    Zarathustra[H] Official Forum Curmudgeon

    Messages:
    25,279
    Joined:
    Oct 29, 2000
    That is too bad.
     
  28. Zarathustra[H]

    Zarathustra[H] Official Forum Curmudgeon

    Messages:
    25,279
    Joined:
    Oct 29, 2000
    Ah,

    Yeah, as I mentioned before, I tried using a couple of Brocade BR1020 cards with transducers and OM3 cable from The Fiber Store before I went 10GBaseT.

    Had so many problems with them that it made me swear off ever touching fiber again.

    While I like enterprise hardware, I am very much not an Enterprise user though.

    For home and small business I think 10GBaseT is a great solution, if it only came down to affordable prices. There power and long run cable sensitivity is less of an issue.

    I have no doubt fiber is king in large scale enterprise implementations.
     
    Lakados likes this.
  29. Lakados

    Lakados Gawd

    Messages:
    654
    Joined:
    Feb 3, 2014
    But just about every switch you can get has 2 or more SFP+ ports on it, most new servers don't even have a 10G option but they do have SFP+ ports as options. You can buy 3m SFP+ patch cables for less than $20, and 1m cables for less than $10, just run those directly from the switch to the server. Yeah you loose backwards compatibility with that but you still have the option of just running a standard cable from the switch to one of the existing 1GB ports on the server should the switch port fail for some reason. You could always just run the cable anyways program the static and disable the port then configure it as a fail over in the event the SFP+ connection fails.
     
    ZeqOBpf6 likes this.
  30. Lakados

    Lakados Gawd

    Messages:
    654
    Joined:
    Feb 3, 2014
    Yeah I can't say I have ever met a Brocade I liked, I have had really good experiences with the StarTech PEX20000SFPI, I have a number of them installed in my older servers and they have been solid.
     
    Zarathustra[H] likes this.
  31. prb123

    prb123 n00bie

    Messages:
    3
    Joined:
    Dec 23, 2011
    Lakados likes this.
  32. Lakados

    Lakados Gawd

    Messages:
    654
    Joined:
    Feb 3, 2014
    Those are nice switches, but even used they are close to $1000 for the setup you listed there .... a far cry from the sub $300 most are hoping for. But yeah if I had to go with 10GBase-T for some reason that would be the product line of my choice.
     
  33. arnemetis

    arnemetis 2[H]4U

    Messages:
    2,095
    Joined:
    Aug 2, 2004
    This thread has had me start looking at other options lol. I think I'm shifting towards doing some kind of sfp+ run between the server and my pc, skipping the switch as I really only need the speed between these two pcs. Now the trouble is finding what I actually need, in a length that will work for me. Doesn't look like they have keystone adapters for these. But that's getting off topic, I need to do more research and maybe make a thread over in networking.
     
  34. toast0

    toast0 Gawd

    Messages:
    826
    Joined:
    Jan 26, 2010
    Softlayer uses it in their newer deployments. The runs are short though, that's just for rack switch to servers. Seems to work fine, and I think it's easier to get port speed negotiation going (they charge more for 10g than 1g, even though it's the same hardware either way -- i think they sell 100M for cheapskates too; but my boss isn't that cheap, we get 2x1G for most things and 2x10G for the stuff that needs it)
     
    /dev/null likes this.
  35. Crotan

    Crotan Gawd

    Messages:
    914
    Joined:
    Jan 2, 2007
    10 gig RJ45 nics have also come down in price significantly.

    https://www.amazon.com/Aquantia-NIC-5-speed-Ethernet-Network/dp/B07B3G4S4J?th=1

    Aquantia is the same company behind all of multgig asics/nics in Cisco's switches, and you could get these for $59 on black Friday. So we've come a long way.
     
    Lakados likes this.
  36. Lakados

    Lakados Gawd

    Messages:
    654
    Joined:
    Feb 3, 2014
    Leviton will have any cable or wall plate or keystone you can imagine.
     
  37. arnemetis

    arnemetis 2[H]4U

    Messages:
    2,095
    Joined:
    Aug 2, 2004
    Yeah and monoprice actually has one too, so it's not as unheard of as I though, thanks. This weekend I'll get some measurements the the distances I need and make a post over in networking.
     
  38. Burticus

    Burticus 2[H]4U

    Messages:
    3,226
    Joined:
    Nov 7, 2005
    I wouldn't mind some RJ45 10g for the house, but honestly except for full system backups or massive file transfers, 1g is plenty. Hell, 100mb is fine for web surfing and gaming.

    At work we use multi mode fiber OM3 for the 10/16/40g runs to the fiber switch (Brocade) going from server to SAN. The 10g trunk runs are also OM3 fiber but I think that's more of a legacy thing, I've seen new equipment coming in with 10g RJ45 jacks.

    It wasn't that long ago (2004 era) that 100mb full duplex was considered optimal for datacenter servers, and only the really high priority stuff got gigabit.

    For the longest time I used to collect all the fiber we tossed thinking it was worth something or I'd use it some day. 15 years later a giant hefty bag of it hit the trash.

    So I welcome our new super fast ethernet overlords... as long as I can keep using the endless supply of Cat5e and Cat6 I have rescued from dumpster runs.

    edit - looks like the 10g rj45 cards are $100... couldn't find a switch under $400 on amazon in the 60 seconds I spent looking.
     
  39. cdabc123

    cdabc123 [H]ard|Gawd

    Messages:
    2,038
    Joined:
    Jun 21, 2016
    Just a FYI I'm not sure if they have shipped yet but the tallos 2 workstations (power 9 cpu) have pcie 4
     
  40. Zarathustra[H]

    Zarathustra[H] Official Forum Curmudgeon

    Messages:
    25,279
    Joined:
    Oct 29, 2000

    There's a Talos 2? I'm going to have to tell Morgan!