![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
- Joined
- Dec 19, 2005
- Messages
- 18,526
“Now that Blackhole chips are available, Tenstorrent is working on building successively bigger training clusters. So far, a training cluster of 6 Backhole Galaxies (192 chips) has been built, with bigger clusters coming over the next six months. The eventual aim is a data plane engine of 16 Galaxy servers, with another 16 Galaxies for switching (Tenstorrent uses its own chips as switches) and another 16 as an optimizer. This includes a certain amount of redundancy.
“The demo is going to be [the cluster] running and then you pull any cable out and it doesn’t stop running,” he said. “In principle, we could turn off any server we want and it would keep running.”
Korean company Moreh is building a training stack for Tenstorrent hardware, but that is for their own product, Keller said. The stack for Tenstorrent’s own training clusters is being developed in-house.
Fast inference can be achieved with four Galaxies (128 chips), Keller said, noting that reasoning is due to require a million times more tokens versus today’s LLM workloads (the real upper bound on the amount of inference compute required is the limit on how many unique questions humans can come up with, he said). Even though inference is likely to be the bigger market in the long term, Keller firmly dismissed any suggestion that the company would or should focus entirely on inference.
“I have a mission to radically lower the cost of training,” he said. “The fun challenge is how do you become a proper development platform to do new things, I’m really interested in that. How do you do new things if you can’t train?”
Training is an essential part of the stack, just like building an in-house CPU is, Keller said.
“I want to build great AI computers,” he said. “That’s the problem with doing part of a thing. [If we focused on serving models from the cloud, for example], what if somebody wants to develop models, where they want to put their own AI technology in their own product? I’m interested in that bigger picture.”
This is despite the fact that training and inference are separate, uniquely difficult problems to solve—especially at scale.
“Everybody says AI hardware is just a matrix multiplier with some software—that’s the stupidest thing I’ve ever heard,” he said. “Yes, there’s a matrix multipler. Yes, there’s some software. But making it work at scale…”
The biggest version of the training computer Tenstorrent is currently planning will have two million RISC-V cores operating in parallel, programmed from a single program.
“AI is stylized, it’s not general-purpose HPC programming, but it’s still fairly dramatic, co-ordinating millions of processors from a simple program with tens of people is pretty good,” Keller said.
“We are going to build stupidly big computers,” he added. “It’s really fun.””
Source: https://www.eetimes.com/jim-keller-whatever-nvidia-does-well-do-the-opposite/
“The demo is going to be [the cluster] running and then you pull any cable out and it doesn’t stop running,” he said. “In principle, we could turn off any server we want and it would keep running.”
Korean company Moreh is building a training stack for Tenstorrent hardware, but that is for their own product, Keller said. The stack for Tenstorrent’s own training clusters is being developed in-house.
Fast inference can be achieved with four Galaxies (128 chips), Keller said, noting that reasoning is due to require a million times more tokens versus today’s LLM workloads (the real upper bound on the amount of inference compute required is the limit on how many unique questions humans can come up with, he said). Even though inference is likely to be the bigger market in the long term, Keller firmly dismissed any suggestion that the company would or should focus entirely on inference.
“I have a mission to radically lower the cost of training,” he said. “The fun challenge is how do you become a proper development platform to do new things, I’m really interested in that. How do you do new things if you can’t train?”
Training is an essential part of the stack, just like building an in-house CPU is, Keller said.
“I want to build great AI computers,” he said. “That’s the problem with doing part of a thing. [If we focused on serving models from the cloud, for example], what if somebody wants to develop models, where they want to put their own AI technology in their own product? I’m interested in that bigger picture.”
This is despite the fact that training and inference are separate, uniquely difficult problems to solve—especially at scale.
“Everybody says AI hardware is just a matrix multiplier with some software—that’s the stupidest thing I’ve ever heard,” he said. “Yes, there’s a matrix multipler. Yes, there’s some software. But making it work at scale…”
The biggest version of the training computer Tenstorrent is currently planning will have two million RISC-V cores operating in parallel, programmed from a single program.
“AI is stylized, it’s not general-purpose HPC programming, but it’s still fairly dramatic, co-ordinating millions of processors from a simple program with tens of people is pretty good,” Keller said.
“We are going to build stupidly big computers,” he added. “It’s really fun.””
Source: https://www.eetimes.com/jim-keller-whatever-nvidia-does-well-do-the-opposite/
