CPU with HBM?

[Bandalo]

I've heard AMD might be working on APUs with HBM, but would this work for a mainstream CPU? If Nvidia can fit big pascal + HBM on a single interposer for the big Pascal cards, why not a 6700k core + 16GB of HBM?

What would the limiting factor be? Cost? Thermals? Obviously you wouldn't be able to upgrade the memory, but would the advantages of speed be worth it?

 

[defaultluser]

Cost. From what I've heard, the interposer and stacks of ram have a high failure rate. This is why I'm fairly certain the "rumors" of HBM on AMD integrated graphics is just fanboy delusions. But AMD does need to come up with a way to fix the bandwidth issue.

If HBM didn't have such issues with cost, it would be in GPUs that don't cost over four hundred dollars. The Fury X gets rolled by the GTX 1070 in most games AND has half the memory, and yet AMD can't sell it for less than $400 and still make money. The massive 512-bit bus on the 290X was cheaper to implement, and it had just as many chips. You can be sure AMD was making plenty of money from that card at $330, as they almost posted a profit last quarter.

Intel doesn't use HBM because they don't need that kind of bandwidth on the majority of their processors. And for those that do, they have the L4 cache. So Intel already solved the problem that HBM attempts to solve. Their fast 128MB DRAM L3 cache is cheaper because it's just mounted on-package - no complex interposer required.

The other problem HBM has is locking down OEM memory SKUs. This already happens for high-priced ultra-portables, but imagine if it happened for desktops and servers too. You can get some flexibility due to failed stacks (like P100 12GB graphics card), but you still have to stock every power of two memory configuration your customer could want to buy, or you lose the purchase (4, 8, 16, 32, 64GB for Skylake processors).

With sticks, you just plug in what you want.

 

[Bandalo]

Well, since the tech is new, it's going to be more expensive. But both AMD and Nvidia have said they'll be moving to HBM designs in the future. Once they start coming out in volume, I'd imagine the costs would have to come down considerably.

Intel went to their L1-2-3 cache scheme because HBM didn't exist. Now that it's around and available, I'd wonder if there would be performance gains to be had for changing that structure and using HBM2/3 instead. If you have fast access to the huge bandwidth HBM can provide, you wouldn't need L3 cache (or wouldn't need as much). At least I'd imagine that would be the case, but I might be missing something in the equation. I know back in the day, you had external cache chips installed between your RAM and your 486, then Intel moved the cache on-die...why continue the trend and move ALL the RAM?

I could see this being popular on the desktop market (if it does give a performance boost anyway) and laptops. It'd save a lot of space on the motherboard for laptops if you didn't need room for the memory external to the CPU. Servers usually want far more memory, so I'm not certain where the ceiling is for HBM capacity.

 

[defaultluser]

Yeah, I'm concentrating on today. Today, HBM is nowhere near ready. Call me back when $200 graphics cards feature the new memory tech, and I'll start badgering AMD and Intel (and all the game consoles) to adopt it

It would be nice for laptops to have the extra bandwidth in even smaller form-factors, but I think we're already hitting the thermal limits of these tiny designs. So we're not bandwidth-limited. And Moore's law is hitting a wall, so we can only improve performance/watt so much. It's a fantastic dream, just not really reality.

 

[Bandalo]

Yeah, the question would really be "is there an advantage for CPUs to use HBM". If it gives you a 20% performance increase, then it seems worth it, since HBM seems better (or at least not worse) on power consumption. And the cost will come down over time and with volume. But if it adds a lot of cost and complexity for no measurable performance gain, then it'll never happen.

 

[defaultluser]

Yeah, the question would really be "is there an advantage for CPUs to use HBM". If it gives you a 20% performance increase, then it seems worth it, since HBM seems better (or at least not worse) on power consumption. And the cost will come down over time and with volume. But if it adds a lot of cost and complexity for no measurable performance gain, then it'll never happen.

Well, servers would love the extra bandwidth of HBM. There's a very good reason they have four channels of ram per-processor (capacity AND bandwidth). But again, they wouldn't love the "must be installed at purchase time." limitation of HBM.

The density would have to go way up as well, since your typical Xeon CPU can get 512GB, and future chips get 768GB of ram PER SOCKET.

So really I just see desktop/mobile chips making the transition. If anything happens on the server side, it will be large caches made from HBM...but not sure if that will be any cheaper or faster than Intel's eDRAM cache.

But thanks to DDR4 I don't see the bandwidth need for HBM on desktop/laptop coming anytime soon, so the push for HBM is not happening anytime soon.

 

[Shintai]

I've heard AMD might be working on APUs with HBM, but would this work for a mainstream CPU? If Nvidia can fit big pascal + HBM on a single interposer for the big Pascal cards, why not a 6700k core + 16GB of HBM?

What would the limiting factor be? Cost? Thermals? Obviously you wouldn't be able to upgrade the memory, but would the advantages of speed be worth it?

And that APU will cost what? You are not going to see a consumer APU with HBM. The APU in question is a theoretical 300W APU to compete against Xeon Phi.

HBM only got 2 advantages left at the moment. Size and ECC.

Xeon Phi uses HMC and its more likely to be used by Intel than HBM. HMC is also better suited for servers. HMC doesn't require an interposer either.

At the current time the future of HBM is questionable. They are struggling to reduce cost, but that will further degrade the product. And future revisions are many years out.

Since you mention Nvidia. You know GP100 uses HBM2. It does so mainly for ECC, secondary size. But Nvidia tells you a big thing, when its cheaper for them to spend ~100M$ on the GP102 design, just to remove FP64 and use GDDR5X instead.

DDR5 and GDDR6(Samsungs name for GDDR5X) are also coming now.

But in short the downside of HBM is cost, and its massive. Even if you imagine Intel and AMD using it for all CPUs/APUs. Lets say 300M CPUs a year with a ~5$ average interposer. That's 1.5B$ alone there, just for an interposer with static cost. To compare, a 128MB EDRAM is less than 3$. And then we can start to talk about TSV cost, product loss when one step goes bad, HBM memory cost.

 

[defaultluser]

Since you mention Nvidia. You know GP100 uses HBM2. It does so mainly for ECC, secondary size. But Nvidia tells you a big thing, when its cheaper for them to spend ~100M$ on the GP102 design, just to remove FP64 and use GDDR5X instead.

One small observation here: pure compute loads can use a whole lot more bandwidth than your average game render, especially because you're not working with compressed attributes (i.e. textures or buffers). And ECC eats up a sizable chunk of available memory AND bandwidth, making HBM even more important.

Is the ECC Performance Price Worth it for GPUs?

Yes, you could make it work with GDDR5X, but it would not be enough bandwidth for demanding users.

 

[SomeGuy133]

Well, servers would love the extra bandwidth of HBM. There's a very good reason they have four channels of ram per-processor (capacity AND bandwidth). But again, they wouldn't love the "must be installed at purchase time." limitation of HBM.

The density would have to go way up as well, since your typical Xeon CPU can get 512GB, and future chips get 768GB of ram PER SOCKET.

So really I just see desktop/mobile chips making the transition. If anything happens on the server side, it will be large caches made from HBM...but not sure if that will be any cheaper or faster than Intel's eDRAM cache.

But thanks to DDR4 I don't see the bandwidth need for HBM on desktop/laptop coming anytime soon, so the push for HBM is not happening anytime soon.

its not just bandwidth its about energy size and latency.

Memory cube and HBM will allow pocket size desktops down the road when tech becomes main stream. It will not be far fetch in 5 years or less to make a 6700K (modern 4 core like cannon lake or something else) 32GB HMB/MC with a decent integrated GPU on a single SOC that can sacle with power source and cooling to fit into a device the size of your cell phone but maybe half inch thick. Plus nano sphere batteries are on the horizon....some time. They were supposed to be out 2012 but whatever.

Some how my 4.8GHz 6700K only uses 55 watts amke that cannon lake and we are looking at 40 watts maybe with full load. Pocket sized full blown desktops here we go!

 

[Shintai]

HBM2 doesn't have any energy benefits. It had vs GDDR5, but it lost it vs GDDR5X/GDDR6. Also in terms of density, it could never compete with DRAM. And using 16GB some 2-3W becomes 20W, that's anything but what the mobile and server wants, unless the benefits are so massive it can make up for it. Latency there isn't anything to get either.

The question always comes back to, are people willing to pay for it. And are those willing a big enough market to make it worthwhile. The answer is no. The first step would be EDRAM caches on a 6700K/7700K type CPU.

 

[Bandalo]

I'd seen the power comparisons between HBM and GDDR5X, but never between HBM and something like DDR4.

 

[Bandalo]

HBM only got 2 advantages left at the moment. Size and ECC.
....

But in short the downside of HBM is cost, and its massive. Even if you imagine Intel and AMD using it for all CPUs/APUs. Lets say 300M CPUs a year with a ~5$ average interposer. That's 1.5B$ alone there, just for an interposer with static cost. To compare, a 128MB EDRAM is less than 3$. And then we can start to talk about TSV cost, product loss when one step goes bad, HBM memory cost.

That really addresses the question. I know HBM has been touted as the "next best thing" for GPUs, and the internet is full of people saying they won't buy a new GPU without HBM. But the question really is how much faster would a well-designed CPU/HBM combo really be versus the existing designs? The size advantage is probably not relevant in the desktop or server market, but may be in notebooks/portables. ECC would be great for servers, but not relevant for desktops.

The cost now is huge, and I get that. But the cost will come down with time. And you have to consider the whole system cost as well, not just the CPU & memory. If I build a desktop motherboard that is designed for HBM + CPU, then I don't need memory slots, I don't need all the additional traces, my motherboard design and validation just became far cheaper, which would likely offset or cancel out the cost of the interposer.

Over the years Intel has incorporated the GPU, secondary cache, various controllers, and even the voltage regulator on-die. Why NOT the system RAM?

 

[Quix]

We might see something like that, but it's going to be on the top-end servers first and then slowly trickle down. It won't be available on consumer chips for years. Also it will likely kill socketed chips, soldered only.

 

[Bandalo]

I agree, we're probably a few chip-generations away from this type of design. Not sure why it would kill sockets though, seems like it could easily be a socketed chip. In fact, you'd need far fewer pins on the socket since you don't need to connect to external memory. IIRC, about half the pins on a X99 socket are for memory...

 

[pxc]

Other than helping the IGP and boosting performance by a few extra percent, on package memory hasn't really been a huge boost so far. The change of "Crystal Well" on Skylake-R to behave as a main memory buffer (as opposed to an L3 victim cache in older architectures), suggests that Intel is approaching a HBM-like optimization in a different way than rumors of what AMD may do on some upcoming processors.

Since the eDRAM on Skylake-R is between the system agent and IMC now, it has some of the benefits of HBM since any device on the (PCIe) bus, and of course the CPU, has access to it. In a way, it's integrated into the system as an implicit part of (flat) memory but is much faster than main memory. It's probably going to take demand from applications running into limits before Intel increases the size of the eDRAM, or adds it to more models. (I hope multiple AVX-512 units are added to desktop processors à la a Xeon Phi core, but who knows.)

I think it's a huge mistake for Intel to see eDRAM as a huge profit center, where each die is worth $50-$100. The volume has to be pretty small at that high price.

 

[Shintai]

That really addresses the question. I know HBM has been touted as the "next best thing" for GPUs, and the internet is full of people saying they won't buy a new GPU without HBM. But the question really is how much faster would a well-designed CPU/HBM combo really be versus the existing designs? The size advantage is probably not relevant in the desktop or server market, but may be in notebooks/portables. ECC would be great for servers, but not relevant for desktops.

The cost now is huge, and I get that. But the cost will come down with time. And you have to consider the whole system cost as well, not just the CPU & memory. If I build a desktop motherboard that is designed for HBM + CPU, then I don't need memory slots, I don't need all the additional traces, my motherboard design and validation just became far cheaper, which would likely offset or cancel out the cost of the interposer.

Over the years Intel has incorporated the GPU, secondary cache, various controllers, and even the voltage regulator on-die. Why NOT the system RAM?

The problem is cost isn't going down. And now they are trying to do a reduced product to try and combat that with a lot worse metrics than the original. But its never going to solve the cost issue. TSV, manufactoring faults that ruins all 3 parts etc is never going away. The interposer alone kills it for the cost oriented products. PCB and traces cost pennies compared. Remember, HBM increases power draw as well. You could just as well sell GDDR5X/GDDR6 in DIMMs.

HBM products would also be fixed SKUs. That's a nightmare of its own. Specially since HBM2 doesn't scale well.

For mobile, HBM is also out of the question. It would simply take way too much power and 15W SKUs become something like 28-35W SKUs.

When you ask why not system memory integration. Its about cost and benefits.

 

[Shintai]

We might see something like that, but it's going to be on the top-end servers first and then slowly trickle down. It won't be available on consumer chips for years. Also it will likely kill socketed chips, soldered only.

For servers it would be cache and not system memory. Just as Xeon Phi.

 

[Bandalo]

The problem is cost isn't going down. And now they are trying to do a reduced product to try and combat that with a lot worse metrics than the original. But its never going to solve the cost issue. TSV, manufactoring faults that ruins all 3 parts etc is never going away. The interposer alone kills it for the cost oriented products. PCB and traces cost pennies compared. Remember, HBM increases power draw as well. You could just as well sell GDDR5X/GDDR6 in DIMMs.

HBM products would also be fixed SKUs. That's a nightmare of its own. Specially since HBM2 doesn't scale well.

For mobile, HBM is also out of the question. It would simply take way too much power and 15W SKUs become something like 28-35W SKUs.

When you ask why not system memory integration. Its about cost and benefits.

Cost isn't going down as fast as they'd like, but I'm sure overall cost IS going down. It just has to with the volume being produced. Might just take longer to mature the processes than they expected. I would disagree with the manufacturing faults point though. While it WOULD cost more per CPU/MEMORY product produced, you would trade a LOT of overall system manufacturing faults with the removal of the memory modules, sockets, traces, and all those extra CPU pins that could be removed. Yes, a CPU/memory package from Intel would cost a lot more per part, but the system manufacturer/integrator wouldn't have to buy memory modules or motherboards with memory slots.

There WOULD be more SKUs, but I don't think Intel/AMD would produce multiple memory SKUs for every single category. 90% of your mobile CPUs could be made with 8GB to meet the general market demand. "Enthusiast" K-series CPUs made with either 16/32GB would meet almost every demand. It'd be a shift from how we're used to doing things, but 99% of the computer-buying population wouldn't care.

Regarding power...I'm honestly not sure how much juice the memory sub-system eats up in a typical mobile system, and I couldn't find anything concrete..just some rough numbers of 2-3W per SO-DIMM/DIMM at DDR4.

 

[Shintai]

Intel Custom Foundry EMIB

Another example that the interposer dreams are dead.

But it still leaves some challenges.

Remember on memory, if you increase speed, you increase power usage. Its not coming for free.

And high speed, high density is a huge problem. 32GB would for example be the peak HBM2 with 4 stacks can do until 2020. And it cant even be done today. HMC is faster and bigger densities tho, so you can get away better there.

 

[Bandalo]

Intel Custom Foundry EMIB

Another example that the interposer dreams are dead.

But it still leaves some challenges.

Remember on memory, if you increase speed, you increase power usage. Its not coming for free.

And high speed, high density is a huge problem. 32GB would for example be the peak HBM2 with 4 stacks can do until 2020. And it cant even be done today. HMC is faster and bigger densities tho, so you can get away better there.

The EMIB solution will be interesting. It's a different technique to be sure, but it could be used for the same purpose, moving RAM into the CPU package and off the motherboard.

Increasing speed by increasing clock rates, yes, you get a definite increase in power use. HBM increases speed by increasing the bus width though, which doesn't correlate to a increase in power. And in some cases, you would use LESS power for the same speed, because you can afford to clock lower. But I do understand your points.

There are challenges to be sure, but it wouldn't surprise me to see something happen in this area in the next 3-4 years. It really depends on the success of Vega and Volta...