Will NPUs soon become useless silicon? — Windows 11’s Local AI is getting GPU support

Lakados said:
Most have significantly less than 30 tops… it’s only the 2024 CPU’s and up that break 30 everything older is either non existent or around 15.
Click to expand...
I feel in most case, for those it is not the GPU access in some microsoft API that will make them useless they never were.

But that does not mean the NPU made by Archality and others will be useless because GPU exist and can be used instead (or those Apple have been making all along and in phones)... there is an amount of inference per expensive rams/watt/expensive node mm2 that they can be beating easily for some part of the inference process, we know that for sure. If they are in anyway useful, they could still be using less watt than the cpu-gpu to do what they do, they never made sense for a desktop with a GPU in the first place, extreme fix and low latency case outstanding.

Beter enough to be worth it for heavy usage, enough for Nvidia to use them and they make the best gpu for it.
 
The only reason the NPU exists is because Microsoft thought AI would be a big part of Windows, and that was a future nobody wanted. You certainly don't need a 50 TOPS NPU for web cams and voice recognition. If given the choice, I'd rather have that silicon for a faster GPU and CPU.
 
DukenukemX said:
The only reason the NPU exists is because Microsoft thought AI would be a big part of Windows, and that was a future nobody wanted. You certainly don't need a 50 TOPS NPU for web cams and voice recognition. If given the choice, I'd rather have that silicon for a faster GPU and CPU.
Click to expand...
NPUs have existed on smartphones since 2018 for many various functions, which is a lot more efficient on the battery than utilizing the GPU or CPU, depending on the AI function utilized.
On x86 laptops/tablets, an NPU definitely will help with webcams, though you are right in that 50 TOPS is totally not needed and 25 or lower could totally perform that function at 1080p easily.
 
I got a new company laptop a Lenovo P6v Gen 3.

Things never used in the laptop:
The NPU of the Core Ultra 7 Processor 265H
The NVIDIA RTX PRO 1000 Blackwell GPU.

The 2400p touch Dolby Vision OLED is nice though.

I'd rather have my GPU in my PC do the "NPU"-liftning, do not need a NPU in my CPU or in my laptop.
 
Armenius said:
Wait, NPUs weren't useless at some point?
Click to expand...
Depends on frequency of use. The whole point of the NPU is to do AI like tasks efficiently, but as far as I can tell it's mainly used for making web cams and voice recognition efficient, which are features I either never use or hardly use. This makes more sense in a cell phone which may also not see these features frequently used, but every ouch of power savings counts and having extra CPU and GPU cores are marginally beneficial for a mobile phone.

The problem is that these NPU's do take up a substantial amount of silicon space that could be better used for a faster CPU+GPU. Look at Intel's Panther Lake where the 50 TOP NPU takes up as much space as the two performance cores, and even slightly more than the four efficiency cores. For power savings that I feel could be achieved if the CPU and GPU were made to better manage it. Especially the GPU which can do 122 TOPS, but just needs a way to limit it's performance down enough to use as much power as that 50 TOP TPU.

Like I said, these 50 TOP TPU's are not meant to make webcams and voice recognition power efficient but to make services like Recall and Copilot more efficient. You know, the two things that people avoid having in their computers.


intel panther lake die shot.jpg
 
DukenukemX said:
The problem is that these NPU's do take up a substantial amount of silicon space that could be better used for a faster CPU+GPU.
Click to expand...
True but NPU big advantage once they get their own chipler, they can use substantial amount of very old node-cheap wafer silicon, many NPU strategy (to retake this NPU startup attempt, https://hardforum.com/threads/welcome-to-archality-ai-inference-by-enthusiasts.2047929/), cerebras, grog, etc... do not scale that well with better node, a bit more cache like than raw compute like in that regard.

Do not let those 30 tops style NPU think they define the future of that space, NPU will be almost certainly used for certain step of local AI workload and a typical local AI workload will be your AI agents using Blender-solidworks to make your 3d to be printed model or video game assets, than code, than who knows.

Do not be surprise if the next GPU after Rubin for desktop will feature grog like chiplet on it, with the GPU coming with its own attached NPU, that will be able to do instant DLSS and others affair. Nvidia NPU on their currently deployed system use relatively cheaper Samsung 4nm SF4X, could make sense in 2030 on the consumer space to still be on that level, grog on old global foundries 14nm node was having nice results.

DukenukemX said:
as far as I can tell it's mainly used for making web cams and voice recognition efficient, which are features I either never use or hardly use.
Click to expand...
There is also inference based upscaling on Laptops that seem to work well (and those laptop monitor tend to have ridiculous high res for gaming relative to the gpu capacity), you trade a virtual zero effect on FPS by not using the gpu to do it in exchange of more latency as the image data need to be moved around more I think. Also has advantage to be able to upscale anything, game does not need any change or support.
 
Last edited:
Marees said:
good rant on NPUs

https://open.spotify.com/podcast-ch...i=ZYiJDFfXTpO_UxVvnComvQ&utm_source=copy-link
Click to expand...

AMD NPU in a xbox and the game engine that use them, will not necessarily be like an Intel laptop NPU doing auto-sr.
On panther lake the NPU is on the CPU tile not the GPU tile having that latency between the 2 (OS, memory system and games are all not made for it either), if it is used for upscaling on a xbox mainly, it could be on the GPU tile or at least I am not sure what they think it would be done the way a Microsoft PC memory system and Intel tile choice limitation create would be present in a custom Xbox soc.

They will have unified memory with the cpu-gpu access to data to do, on the same SOC package, the latency hit in exchange of zero fps hit on an Intel laptop will not be similar here.

And the idea that you would need 2,000 TOPS that for large model talk, yes, but DLSS type model are not global world knowledge general LLM model, DLSS 5 upscaling is an extremelly small 110MB model, much smaller than the very small 1B only local LLM outthere


DukenukemX said:
NPU doesn't need to be as large as a CPU core to do things like web cam image blurring. Even that can be done by the GPU anyway.
Click to expand...
yes but on the next Xbox it could for something like upscaling/frame generation, that you know in advance you will be pretty much always using for the common 4k output and lower power usage for it could be interesting for the available power enveloppe and all games known in advance it is available. NPU are never about need to be done, it is how much watt used while doing it at the moment, the upcoming large sram NPU grok/cerebras/large array of competition like the one here (https://hardforum.com/threads/welcome-to-archality-ai-inference-by-enthusiasts.2047929/, if it happen) will be more about super speed and low power doing it.

They also have predictable continous execution and response time that can be interesting for some application, asic built for inference in the future will be quite the norm seem a reasonable guess, how people call them is up in the air (GPU will have them and peopel will maybe not use different names, simply assume they are there). A bit like matrix math array is taken by granted something GPU do, no need it to call it Tensor/WMMA units cores or care how they are called.
 
My Core 265H NPU has ~30 TOPS
My RTX Pro 1000 has ~570 TOPS

So less than 10% of My GPU has more TOPS than my NPU in the latop.
It's better to have a 10% faster GPU than a NPU, NPU's have ver limited use cases.
And moving data between CPU/NPU/GPU is a bad idea,
 
littlegreenmenfromhome said:
My Core 265H NPU has ~30 TOPS
My RTX Pro 1000 has ~570 TOPS

So less than 10% of My GPU has more TOPS than my NPU in the latop.
It's better to have a 10% faster GPU than a NPU, NPU's have ver limited use cases.
And moving data between CPU/NPU/GPU is a bad idea,
Click to expand...
Moving data really isn't too much of an issue. There's a million ways to deal with it especially in a unified memory model. The problem with the GPU is power. NPUs idle at around 1 watt (or less) and even at load max its like 10 watts. The vast majority of users have laptops so it makes sense from that perspective. It feels like the NPUs came about because of Microsoft and the desire to have Pilot always up in your face which means you need something that's low power. 40 TOPs is fine for typical conversational AI annoyance.
 
LukeTbk said:
yes but on the next Xbox it could for something like upscaling/frame generation, that you know in advance you will be pretty much always using for the common 4k output and lower power usage for it could be interesting for the available power enveloppe and all games known in advance it is available. NPU are never about need to be done, it is how much watt used while doing it at the moment, the upcoming large sram NPU grok/cerebras/large array of competition like the one here (https://hardforum.com/threads/welcome-to-archality-ai-inference-by-enthusiasts.2047929/, if it happen) will be more about super speed and low power doing it.

They also have predictable continous execution and response time that can be interesting for some application, asic built for inference in the future will be quite the norm seem a reasonable guess, how people call them is up in the air (GPU will have them and peopel will maybe not use different names, simply assume they are there). A bit like matrix math array is taken by granted something GPU do, no need it to call it Tensor/WMMA units cores or care how they are called.
Click to expand...
If the Next Xbox is using the NPU for something then it's because that's something Microsoft wanted to do, not something they needed to do. You're always better off using the GPU to do GPU stuff. If the NPU was a good idea then AMD and Nvidia would have made good use of it by now. Also, if consumers were asked if they wanted the NPU vs 10% more GPU version of a chip then I guarantee you they'd always go for 10% more GPU. The NPU is not something consumers want.
 
DukenukemX said:
If the NPU was a good idea then AMD and Nvidia would have made good use of it by now
Click to expand...
The nuance between an NPU and an LPU Nvidia (or Google TPU or the new meta training/inference chips they just announced) is using in the upcoming Rubin datacenter can be thin here, in the sense the statement is more about specialized ASIC made from the ground up for certain type of giant volume specific inference/math being used on the SOC or not, the exact strategy and name it will use will vary if it for LLM or imagery, etc...

Apple has made good use of them for about a decade now (A11 bionic was 2017, 0.6 TOPS back in the days... up to 35 now, M1 was 2020), because on mobile using less power to do a task can be quite interesting, console do have something a bit similar in term of fixed modest power budget for what they try to do, if an specialized upscaled ASIC can do "DLSS5" for 12 watt instead of 45 watt on the GPU, it can be nice and it let the gpu run at higher clock making up being a bit smaller with a positive margin, 140 watt a bit smaller gpu can beat the 105 watt bigger one, while having better upscaling/processing that use more compute.

Needed is a strange word to ever use with computers chips, CPU are turing complete, you can always do everything on a cpu, you do not need a gpu (we were getting by and playing video game without those when I was a kid), you want one because you want to do a specific task for cheaper and less watt, same for an NPU when you want one.

The less power constrant you have, the less specialized ASIC for a task become interesting, we go from phone->console/laptop->desktop in that regard.

It is all really new what it will look like in 10 years could be quite different, GPU used for the high parrallel part of AI inference (the pre-fill, tokenisation of input) and some ASIC for the inference could be common. GPU maker will put said ASIC directly on the chip or package of course and people will be free to still call it GPU if they want (and if the marketing focus group tell them that the best option)... that of course irrelevant in the technical sense if they start to say the GPU ship with LPU/NPU cores or not, but that will please or not different crowd.

DukenukemX said:
The NPU is not something consumers want.
Click to expand...
they have not the first idea of what they are and why they use less watt to do what they do versus cpu-gpu, they want the phone that has them because battery life/speed of task that use them is better than the phone without them historically.

Not sure the consumer want the PS6 gpu to have a lot more matrix operation on the gpu instead of an NPU as a strategy to do the same thing particularly, the end result will be what matter here, they will not know how much more WMA inside the GPU was possible here because of the silicon allocation.

Would Xbox still had the money to go the fancy sony way, maybe they would make a nice custom GPU, they will make a giant soc with off the shelve option instead and will hit their margin down and take that pill... not expecting volume to amortize the work of making a custom silicon.
 
Last edited:
You must log in or register to reply here.
Back
Top