PCIe 6.0 Specification: The Interconnect for I/O Needs of the Future

[erek]

You guys get into Webinars?

" In this webinar, attendees will learn more about what is driving the quick transition to PCIe 6.0, including an updated specification release timeline. The webinar will feature a deep dive into the PCIe 6.0 architecture metrics. It will introduce the approach PCIe 6.0 specification is taking to offer new features like PAM-4 encoding and FEC, while preserving its low-latency characteristics required for a Load-Store interconnect and full backwards compatibility. "

https://www.brighttalk.com/webcast/17656/409679

 

[idiomatic]

Run your 4080TI on a single lane.

 

[bonehead123]

Run your 4080TI on a single lane.

And all 16 of my 6th gen 50PB m.2 drives in RAID 800 at the same time, hehehe

 

[/dev/null]

You guys get into Webinars?

" In this webinar, attendees will learn more about what is driving the quick transition to PCIe 6.0, including an updated specification release timeline. The webinar will feature a deep dive into the PCIe 6.0 architecture metrics. It will introduce the approach PCIe 6.0 specification is taking to offer new features like PAM-4 encoding and FEC, while preserving its low-latency characteristics required for a Load-Store interconnect and full backwards compatibility. "

https://www.brighttalk.com/webcast/17656/409679

Run your Quad-SLI GPU on a single lane?

 

[defaultluser]

I think this is the last "quick revision," after they struggled their way through PCIe 3 to 4 (and found enough tricks in the process to open the gateway wide-as-fuck)?

It's going to be another 5-10 years before we see the first x86 chipset support for this. We're ALMOST saturating high-end x4 PCIe 4.0 drives now, (not with sustained bandwidth, but with burst) and every other device is still just fine with x16 PCI 3.0.

Maybe they'll see first use on mobile devices?

 

[Ready4Dis]

I think this is the last "quick revision," after they struggled their way through PCIe 3 to 4 (and found enough tricks in the process to open the gateway wide-as-fuck)?

It's going to be another 5-10 years before we see the first x86 chipset support for this. We're ALMOST saturating high-end x4 PCIe 4.0 drives now, (not with sustained bandwidth, but with burst) and every other device is still just fine with x16 PCI 3.0.

Maybe they'll see first use on mobile devices?

Maybe true, but only needing 1 lane or 2 lanes would reduce complexity on motherboards, or allow more devices. Just thing, pcie 4.0 x8 is just as fast as a full x16 in 3.0. You could run a GPU with half the lanes and not lose any performance, which leaves more for LAN, wifi, USB, thunderbolt, NVME, etc etc. PCIe 5.0 and 6.0 will help even more. Imagine if you didn't have to choose SATA ports or full speed NVME... .

 

[DanNeely]

I think this is the last "quick revision," after they struggled their way through PCIe 3 to 4 (and found enough tricks in the process to open the gateway wide-as-fuck)?

It's going to be another 5-10 years before we see the first x86 chipset support for this. We're ALMOST saturating high-end x4 PCIe 4.0 drives now, (not with sustained bandwidth, but with burst) and every other device is still just fine with x16 PCI 3.0.

Maybe they'll see first use on mobile devices?

Unless they can do something miraculous about power consumption, mobile is completely out of the question. On your AMD 570 board having to stick a fan on the chipset is just an annoyance; in mobile every milli joule of power consumed matters.

The broader lack of PCIe 4.0 on the desktop is mostly about Intel's 10nm faceplant. We should've had it a year or two before when AMD finally delivered. That said I'm not sure we'll ever see this on the desktop either. More expensive PCB requirements (thicker metal layers and much tighter consistency specs) and the need for redrivers to reach the bottom half of the board are responsible for X570 boards being significantly more expensive than x470; and PCIe5.0 is a lot worse on that front. Pre X570 availability estimates I saw were ~$100 extra to a full size ATX board to support 4.0 and closer to $400 extra to support 5.0. The price premium x570 came out at suggests the first estimate wasn't far off. If the second one is in the right ballpark too, 5.0 and higher might become data center only interconnects.

 

[defaultluser]

Maybe true, but only needing 1 lane or 2 lanes would reduce complexity on motherboards, or allow more devices. Just thing, pcie 4.0 x8 is just as fast as a full x16 in 3.0. You could run a GPU with half the lanes and not lose any performance, which leaves more for LAN, wifi, USB, thunderbolt, NVME, etc etc. PCIe 5.0 and 6.0 will help even more. Imagine if you didn't have to choose SATA ports or full speed NVME... .

We're not going to do that: switchable lanes adds board complexity, and you don't want to permanently cut lanes (would castrate your upgrade path).

On AMD Zen 2, we already have x4 dedicated lanes for your primary m.2 drive. That gives you another x4 PCIe 4.0 upstream to share with the rest of the chipset devices. I don't think your imagined use case exists (running dual x4 PCIE 4.0 m.2 cards at max speed, while simultaneous transferring data over your 10Gigabt Ethernet network.)

. Having a motherboard with that much shit connected to the soutbridge(that you're actually worried about saving lanes) also implies that you are a corner-case consumer (who spends $3000 on a new desktop system, becasuse obviously you ahve to buy a $600 motherboard to get quality ).

If you need that kind of simultaneous throughput (and tons of PCIe lanes to attch to external devices), they make threadripper motherboard .

 

[defaultluser]

Unless they can do something miraculous about power consumption, mobile is completely out of the question. On your AMD 570 board having to stick a fan on the chipset is just an annoyance; in mobile every milli joule of power consumed matters.

The broader lack of PCIe 4.0 on the desktop is mostly about Intel's 10nm faceplant. We should've had it a year or two before when AMD finally delivered. That said I'm not sure we'll ever see this on the desktop either. More expensive PCB requirements (thicker metal layers and much tighter consistency specs) and the need for redrivers to reach the bottom half of the board are responsible for X570 boards being significantly more expensive than x470; and PCIe5.0 is a lot worse on that front. Pre X570 availability estimates I saw were ~$100 extra to a full size ATX board to support 4.0 and closer to $400 extra to support 5.0. The price premium x570 came out at suggests the first estimate wasn't far off. If the second one is in the right ballpark too, 5.0 and higher might become data center only interconnects.

But the cost of PCIe 4.0 support is falling - AMD's B550 motherboards are going to be priced closer to $100. Once you reduce the cost or testing tools (and you get PCIe 4.0 implementation from 3rd-parties), the cost craters drastically.

I will agree that PCIe 6 will probably have a permanent price increase with increased thickness, but these will still show up in Enthusiast/Threadripper style desktops.

 

[Ready4Dis]

We're not going to do that: switchable lanes adds board complexity, and you don't want to permanently cut lanes (would castrate your upgrade path).

On AMD Zen 2, we already have x4 dedicated lanes for your primary m.2 drive. That gives you another x4 PCIe 4.0 upstream to share with the rest of the chipset devices. I don't think your imagined use case exists (running dual x4 PCIE 4.0 m.2 cards at max speed, while simultaneous transferring data over your 10Gigabt Ethernet network.)

. Having a motherboard with that much shit connected to the soutbridge(that you're actually worried about saving lanes) also implies that you are a corner-case consumer (who spends $3000 on a new desktop system, becasuse obviously you ahve to buy a $600 motherboard to get quality ).

If you need that kind of simultaneous throughput (and tons of PCIe lanes to attch to external devices), they make threadripper motherboard .

No, I understand. My point was it's possible, especially with pcie 5.0 or 6.0. It isn't just for corner cases, it's for selling cheap boards (less traces) or more features... Just look how much faster and how many more ports you get today than a few years ago. Going forward it's going to be more, and things like 10gbe will be common instead of niche. USB 4.0 will require more bandwidth and nvme will be pushing 10GB/s. If you can use less lanes for peripherals it means you can run more things straight to the CPU lanes. I'm not so much saying it will happen with pcie 4.0 but can see possibility going forward.

 

[Balkroth]

Well, we have 2x 100 Gbps nic's out there now, which definitely is more than a 3.0 x 16 slot can handle .

 

[CraigHB]

They're coming out with new PCIe standards way faster than anyone can implement them. Intel is determined to stay on PCIe 3.0 long as possible. The only real benefit for PCIe 4.0 right now is faster hard drives. However the difference in speed for day to day use is debatable. I have it and I like it for large file transfers, but I don't do those all the time. I could get by on PCIe 3.0 no problem and would only be inconvenienced by some slower large transfers here and there.

 

[Ready4Dis]

Well, we have 2x 100 Gbps nic's out there now, which definitely is more than a 3.0 x 16 slot can handle .

Yeah, but it'll be a while before that comes down to us mere mortals in the desktop market. There was a great test a while back with AMD servers and high bandwidth networks, wish I could find it. That said, mellanox (I guess now NVidia since they recently got approved to buy them out?) makes pcie 4.0 100gbe cards (2 ports!).
Here is the blog from mellanox anyways:
https://blog.mellanox.com/2019/08/feeding-the-data-beast-with-pcie-4-0/

What do you do if you have Intel? Just run pcie 3.0 x16 and get slightly slower speeds I presume and only use a single port? I can't beleive this type of stuff isn't at the forefront, even Intel had to do their own product testing using AMD hardware (optane) because they don't have their own. Seems in these higher end situations it could make a huge difference. You can use a single slot, 16 lanes for AMD while if you did the same with Intel you'd use 2 slots and 32 lanes. That's a really big difference for just one thing.

 

[Ready4Dis]

They're coming out with new PCIe standards way faster than anyone can implement them. Intel is determined to stay on PCIe 3.0 long as possible. The only real benefit for PCIe 4.0 right now is faster hard drives. However the difference in speed for day to day use is debatable. I have it and I like it for large file transfers, but I don't do those all the time. I could get by on PCIe 3.0 no problem and would only be inconvenienced by some slower large transfers here and there.

Yeah, for home use you won't really notice much. It would be nice though, for example, those pcie x1 slots would be able to supply 2x the bandwidth (or a x4 slot that works as well as a current x8, or that second video card slot at x8 that can run as well as a 3.0 x16), so a 10gbe on x1 or video capture card, etc. It's not worth building these specialized devices for the few that could actually use them. I think we'll start seeing a bit more once Intel starts supporting pcie 4.0. It opens possibilities, whether we see them come to fruition is entirely different .

 

[DanNeely]

They're coming out with new PCIe standards way faster than anyone can implement them. Intel is determined to stay on PCIe 3.0 long as possible. The only real benefit for PCIe 4.0 right now is faster hard drives. However the difference in speed for day to day use is debatable. I have it and I like it for large file transfers, but I don't do those all the time. I could get by on PCIe 3.0 no problem and would only be inconvenienced by some slower large transfers here and there.

The PCIe group didn't develop anything beyond 3.0 for an extended time because until data center (read initially machine learning) customers started making hardware that needed more than a 3.0 x16 offered there was no need for anything faster. Now that it's a bottleneck for multiple end users (accelerator cards, high speed networking, high end storage vendors wanting 1 lane SSDs to pack more per enclosure) they're iterating as fast as possible to try and catch up.

Intel is stuck on 3.0 not by choice, but due to their ongoing 10nm manufacturing debacle combined with design practices that rigidly locked designs to specific process nodes, meaning their 4.0 designs have been sitting stuck because they can't fab them on 14nm and 10nm isn't available in any significant volume yet.

 

[DanNeely]

Yeah, for home use you won't really notice much. It would be nice though, for example, those pcie x1 slots would be able to supply 2x the bandwidth (or a x4 slot that works as well as a current x8, or that second video card slot at x8 that can run as well as a 3.0 x16), so a 10gbe on x1 or video capture card, etc. It's not worth building these specialized devices for the few that could actually use them. I think we'll start seeing a bit more once Intel starts supporting pcie 4.0. It opens possibilities, whether we see them come to fruition is entirely different .

Ampre is supposed to support 4.0 when it comes out later this year. AMD is supporting it in their Radeon Instinct data center cards; but I couldn't find anything about when they intend to push it down to consumer level cards.

 

[Ready4Dis]

Ampre is supposed to support 4.0 when it comes out later this year. AMD is supporting it in their Radeon Instinct data center cards; but I couldn't find anything about when they intend to push it down to consumer level cards.

All of their current cards support pcie 4.0... even the 5500xt why wouldn't their next ones? I wasn't holding onto a leak for it, I don't see them going backwards.

Edit: https://www.thefpsreview.com/2019/1...stantial-performance-bump-on-pcie-4-0-vs-3-0/
Example of speed increase for a slower clocked card using pcie 4 vs 3

2nd Edit: This is why it sucks when it's tested on Intel CPU's... it's hamstrung and shows lower performance if it's paired with a "faster" CPU... if you're GPU limited.

 

[DanNeely]

All of their current cards support pcie 4.0... even the 5500xt why wouldn't their next ones? I wasn't holding onto a leak for it, I don't see them going backwards.

Edit: https://www.thefpsreview.com/2019/1...stantial-performance-bump-on-pcie-4-0-vs-3-0/
Example of speed increase for a slower clocked card using pcie 4 vs 3

2nd Edit: This is why it sucks when it's tested on Intel CPU's... it's hamstrung and shows lower performance if it's paired with a "faster" CPU... if you're GPU limited.

Google's apparently being garbage today. None of the articles I looked at from that or a few other similar searches mentioned PCIe versions aside from one saying AMD was holding it back for pro cards initially.

 

[Ready4Dis]

Google's apparently being garbage today. None of the articles I looked at from that or a few other similar searches mentioned PCIe versions aside from one saying AMD was holding it back for pro cards initially.

No problems, now you know why nobody is asking or talking about it for AMD, lol. You can run their (lower end) GPU at x8 and get the same speeds as Intel with x16.. or as my link shows, you can run it at 4.0 x16 on AMD and get even more performance. Higher models don't see as much benefit as they have more ram so less data over pcie while running. So results can vary from no difference to +20 fps as seen in the above link.

 

[DanNeely]

All of their current cards support pcie 4.0... even the 5500xt why wouldn't their next ones? I wasn't holding onto a leak for it, I don't see them going backwards.

Edit: https://www.thefpsreview.com/2019/1...stantial-performance-bump-on-pcie-4-0-vs-3-0/
Example of speed increase for a slower clocked card using pcie 4 vs 3

2nd Edit: This is why it sucks when it's tested on Intel CPU's... it's hamstrung and shows lower performance if it's paired with a "faster" CPU... if you're GPU limited.

Do you know of any sites that've looked at that in depth? My initial assumption of why the 4gb card is seeing much larger gains from PCEe4 than the 8gb card is that Far Cry needed slightly more than 4gb of vram; and the faster bus reduced the penalty from having to swap to main memory. If that's the case though, I'd be concerned that the 4gb card might age poorly on 4.0 systems with the performance margin collapsing once vram requirements grow enough that the 4.0 bus becomes a bottleneck for swapping like the 3.0 one was. Going to the source article and looking at the asassins creed results where the 4gb 4.0 card ended up with results midway between the 8gb cards and 4gb 3.0 one suggests it's already starting to happen in some games; but I'd need data about the amount of vram the games are using to check my theory that doesn't appear to be present in that article.

 

[Ready4Dis]

Do you know of any sites that've looked at that in depth? My initial assumption of why the 4gb card is seeing much larger gains from PCEe4 than the 8gb card is that Far Cry needed slightly more than 4gb of vram; and the faster bus reduced the penalty from having to swap to main memory. If that's the case though, I'd be concerned that the 4gb card might age poorly on 4.0 systems with the performance margin collapsing once vram requirements grow enough that the 4.0 bus becomes a bottleneck for swapping like the 3.0 one was. Going to the source article and looking at the asassins creed results where the 4gb 4.0 card ended up with results midway between the 8gb cards and 4gb 3.0 one suggests it's already starting to happen in some games; but I'd need data about the amount of vram the games are using to check my theory that doesn't appear to be present in that article.

I don't, sorry. Wish someone would do a real comparison, but like I said, my assumption was same as yours. An 8gb card probably wouldn't notice much difference if it was will in vram as it doesn't need to go through pcie to get data. Sadly nobody really bothers testing pcie 4.0 unless it's an NVME benchmark that's useless in 99% of cases. 20% FPS (and better 1% lows) in a game sounds a bit more useful. It can make the difference between barely playable and easily playable. Im sure once Nvidia finally releases pcie 4.0 we'll start seeing comparisons.

 

[IdiotInCharge]

20% FPS (and better 1% lows) in a game sounds a bit more useful. It can make the difference between barely playable and easily playable.

The real problem with this is that it applies largely where the needs of the game outstrip the available VRAM by a bit; by a lot, and performance will tank, and next to a GPU with enough VRAM, performance will be significantly better.

Not a feature to turn your nose up at, but if you could spend +US$30 on a better board or a GPU with more VRAM, for example, which would be the better buy?

 

[Ready4Dis]

The real problem with this is that it applies largely where the needs of the game outstrip the available VRAM by a bit; by a lot, and performance will tank, and next to a GPU with enough VRAM, performance will be significantly better.

Not a feature to turn your nose up at, but if you could spend +US$30 on a better board or a GPU with more VRAM, for example, which would be the better buy?

https://www.pcgameshardware.de/Rade...pecials/PCI-Express-3-vs-PCI-E-4-GPU-1339415/ - Original source, not English, but charts are easy to read, just click on game title to switch between.

That's a good point, but if I'm buying a B550 anyways, I'm not actually spending any extra on a board. Obviously this hasn't been the case (and still won't be for at least a few more days) and I understand your point. To me it's interesting though, but from a financial standpoint in todays market, you're probably right . The 8gb version did see up to 16% minimums/15% average (and as little as margin of error) gain from pcie 4.0 as well.

https://www.techpowerup.com/review/nvidia-geforce-rtx-2080-ti-pci-express-scaling/3.html - Chart I'm refering to is on this page
https://www.techpowerup.com/review/nvidia-geforce-rtx-2080-ti-pci-express-scaling/6.html - Summary showing the 2080ti actually gains more @ 1080p (less memory constrained) than it does at 4k, so opposite of what I would have thought.

You don't lose pcie scaling completely, it's just less (and very dependent on game). Even on a 2080ti there is still good pcie scaling in many games, although obviously no tests for pcie 4.0, but there is a noticeable difference between 3.0 x4, x8 and x16 even at 1080p where one would assume everything fits into vram and it's highly dependent on the game. Example @ 1080P from that link, Assassins Creed Origin x4 gets 92.7 FPS @ x4, and 104.7 @ x8, and 114.5 @ x16... as you can see, the frame rate increase each time it doubled went up about 10fps. If it had support for pcie 4.0, it'd be interesting to see if this scaling continued. My guess is it would have been similar gains. The gains in this game for a lower model card would likely have been more if it had been memory constrained. It's weird (please visit the link) how some games have gains like this and some are almost flat between them. Not sure if it has to do with the way memory is handled by the game or how many (and how fast) calls are being sent through the pcie lanes, or what the difference is. Maybe someone has a chart with used vram for each of these games and that would help with the story.

Alas, after doing some more research:
https://www.techpowerup.com/review/pci-express-4-0-performance-scaling-radeon-rx-5700-xt/23.html - Summary showing, well, not much honestly. pcie 2.0 x16 is almost the same as pcie 3.0 x16 which is pretty much the same as 4.0 x16.

It seems the difference in PCIE speeds for AMD 5700XT cards is not typically as much as nvidias highest end oddly enough. Maybe they are doing memory management differently? Maybe the 5700XT is less memory constrained and more performance constrained? Or one was on Intel and one was on AMD board (So different CPU/MB implementation and CPU constraints?) Hoping someone will put the effort to test these things out when ampere comes out with pcie 4.0 support and/or Intel finally start releasing a CPU with pcie 4.0 support.