Nvidia's Project DIGITS

longblock454

longblock454

2[H]4U
Joined
Nov 28, 2004
Messages
2,779
Specs so far:
- 128GB of unified coherent memory
- can run up to 200-billion-parameter LLMs
- Two Project DIGITS AI supercomputers can be linked to run up to 405-billion-parameter models.
- Available in May from NVIDIA and top partners, starting at $3,000
- Nvidia Press Release Link


Looks to be by far the cheapest way to get 128G of "VRAM", vs buying say some eBay A6000s.

One of these likely in my future.
 
It seems pretty awesome to me. I'll probably get someting like this in the future. This is way more efficient than maxing out a power circuit on a box full of GPUs.

There's already a market for people that want to run all the AI stuff on their own hardware so they don't have to worry about privacy or censorship and can have things tailored specifically to what they want. And this market is going to grow a lot as AI gets better.
 
This is actually awesome. This would be an cool thing to have that you just log into via your web browser on your main PC, private LLM, without needing to use your PC resources.

I'm sure people will release LLM projects on github specifically for this device, so eventually it will be something anyone could use without needing to build a LLM.
 
Dang so this is 3k for the whole system? Or just the gpu hardware? Ither way that is insanely cheap for 128gb vram!

I have been working on my own ai just built a fairly solid rig for it
Amd ryzen 9 7900x
64 gb ddr5 corsair vengeance 6400mts
MSI - MAG B650 Tomahawk wifi mother board
4th samsung 990pr ssd
toped offf with a radeon instinct mi50 16gb
with msi aio 240mm rad version
in a corsaid 4000d.

It does quite well using ollam api backend i use the qwen 500m param model for speed. Mostly the llm just needed to be able to process the context it is given from webscrapes and generate a response. The main magic happens with my code and prompt engineering and prompt templates etc.
I am going to make a scratch model eventually im waiting on some more fans to get here before i do that i need to feed my gpu more air as training a model could take it close to thermal limit and keep it there for a good long time. but man a system like nvidia project digits would really take my ai to the next level to say the least.
 
Oh i need to come up with 3k by may then .... LOL
then my ai rig becomes my gaming rig im sure i can figure out someway to boost gaming performance with an mi50 ... :)

But using qwen .5b as my processing model through ollama gives me very fast responses in my discord (how i mostly interact with the ai)
And qwen when asked to write a story of x words or other question with a long answer generates about 250 tokens per second on the current hardware. On this monster of nvidia's that would probably be a few 1000 tokens per second LOL

I do not know why the hell i stopped coming here and why i did not think to come here for info and or assistance related to my ai project. I mean you can see how long ago i joined lol i was here when water cooling was just starting to really take off LOL When various uses started making water windows for their water cooling res. Oh yeh direct die water cooling blocks made with drills and files LOL Fun times !!!
 
Yeah, I am absolutely going to buy one of these if I can find one
 
yeh that is likely what will screw us over lol people will snap these up then try and price gouge with them. They seem like they are goign to crush it the only thing im a little off put by is the ddr5x ram instead of hbm2 ram hbm2 has much higher bandwidth it is what my mi50 uses and the ram is shared between cpu and gpu hardware similar to a integrated igpu just that in this case the igpu is just bonkers. Now the connection between the ram the gpu and the cpu is going to be much much faster hor ever i believe the ddr5x is going to bottle neck the gpu. Now that said my mi50 is older obviously and there are 5 or 6 higher end ai specific gpus out. So this will be faster than my mi 50 but would it be faster than say a amd ryzen 9 9950x3d with 125 mb ddr5 6400 to 7000mts ram and say 2 mi100 32gb cards for 64gb of hbm2 vram? With nvidias 50series about to drop 4090s and higher end ai dedicated gpus will be dropping in price so 2x mi100s or nvidias equiv will probably come down so you could do 2 of those along with the ryzen 9 config mentioned for aboutt he same3k price point.

My ai is going to be far more than just a hobby. Now not to say that nvidia digits is not a game changer because it absolutely is. This is going to put AI on the desktop of well anyone who wants it. They are aiming the product at the researchers and the like but the fact is they will have a hard time keeping up with the demand from basically everyone like those on the forums here enthusiasts are going to snatch these suckers up faster than free 256 mb thumb drives in 05...
What is hilarious is i was never able to score 1 of those drives back then LOL
I got tons of other good stuff like 250 count dvd rw spindles for free even high speed dvd roms and various contests hell even got a few 128mb vido cards from contest but never 1 single 256mb thumb drive LOL
 
Yes, the memory will be slower than on a traditional high-end GPU right now, from the GPU's perspective.

I wonder whether they will make a rackmount version.
 
Of the digits highly unlikely there is no need and also this thing is smaller than a mac mini and looks to be passively cooled. Now the gpu itself i am sure they will. Everything about nvidia digits seems like it is a stripped down version of their high end AI cluster server hardware. They are looking to put AI in to everyone's hands and let us run with it and take it to the next level. As far as AI companies at the bleeding edge goes there are only a hand full. Because of this is it was just up to them AI would stagnate and possibly even die off there just are not enough people involved in those companies to really take ait to the next level so they are dropping this sort of thing AMD should be dropping something like nvidia digits soon and for around the same price point. I think amd will do something a little more upgradeable at the cost of size.
Intell is to new to the gpu market to be able to make a "competing" product yet. Give them another year though and i think they will be. Intel is working directly with Huggingface and their arc lines of card are already showing good AI performance not at the level of amd and nvidia but they are starting to make some seriously strides in the right direction
 
Bit more detail on digits most is speculative but ....
He mentions interconnecting 2 ore more digits nodes which is what those 2 wide ports are on the back it is how the devices can be connected to each other. I am hoping they release a version with hbm2 memory as it has much higher bandwidth than ddr5x i would pay double for hbm2 variant over ddr5x

View: https://www.youtube.com/watch?v=mjsbvtSEUu4&t=293s
 
You must log in or register to reply here.
Back
Top