![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
- Joined
- Dec 19, 2005
- Messages
- 17,760
HBM5
“HBM memory bandwidth and IO counts are doubling with each new generation, Malik said, with packages growing denser and more complex. Reaching consensus in the industry is challenging, as advances like HBM4 and HBM5 increase IO counts from 2,000 to 4,000, necessitating innovative packaging to accommodate these high-bandwidth connections within limited space.
Malik said Nvidia has not only reduced its product development cycle, allowing it to release a new GPU every year, but it also expects to double memory bandwidth and capacity. JEDEC standards, however, take time. That means Nvidia is opting for a custom solution, he said.
Malik said AI has “blown up” the parameters for capacity. “The top AI models have grown rapidly, and the number of parameters has increased from a couple of million to a couple of billions. We are anticipating a trillion parameters from a single model, where those parameters need to reside in memory.”
He said Marvell’s custom HBM architecture enables a 33% greater memory capacity, as well as up to 25% more area for compute, and a 70% reduction in memory interface power – a metric that’s becoming especially important for modern data centers running AI workloads. The optimized interfaces reduce the required silicon real estate in each die, allowing HBM support logic to be integrated onto the base die.”
Source: https://www.eetimes.com/hbm-innovation-outpaces-standards-development/
“HBM memory bandwidth and IO counts are doubling with each new generation, Malik said, with packages growing denser and more complex. Reaching consensus in the industry is challenging, as advances like HBM4 and HBM5 increase IO counts from 2,000 to 4,000, necessitating innovative packaging to accommodate these high-bandwidth connections within limited space.
Malik said Nvidia has not only reduced its product development cycle, allowing it to release a new GPU every year, but it also expects to double memory bandwidth and capacity. JEDEC standards, however, take time. That means Nvidia is opting for a custom solution, he said.
Growing model sizes demand massive capacity
He said HBM5 will face challenges in standardization and adoption due to architectural complexities, including the placement of the memory and controller. As workloads demand more HBM stacks per compute die, balancing high memory bandwidth with increased capacity becomes crucial for customers designing next-generation AI and compute hardware solutions.Malik said AI has “blown up” the parameters for capacity. “The top AI models have grown rapidly, and the number of parameters has increased from a couple of million to a couple of billions. We are anticipating a trillion parameters from a single model, where those parameters need to reside in memory.”
He said Marvell’s custom HBM architecture enables a 33% greater memory capacity, as well as up to 25% more area for compute, and a 70% reduction in memory interface power – a metric that’s becoming especially important for modern data centers running AI workloads. The optimized interfaces reduce the required silicon real estate in each die, allowing HBM support logic to be integrated onto the base die.”
Source: https://www.eetimes.com/hbm-innovation-outpaces-standards-development/
