-What do you think is the most cost efficient bang for your buck hardware to run AI locally in 2026? From what I've read, multiple GPUs will pool memory together in running local LLMs together unlike SLI/CrossFire days. Probably 2 x RTX 5060 Ti 16GB will be the cheapest poor man's setup?
![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
-
Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.
AI LLM hardware discussion thread
- Thread starter David-Duc
- Start date
depending on what you do-need size wise, used single 3090 or M2 era 24GB mac mini would be alternative...
a single 3090 still have more memory bandwith than 2 modern but 128 bits gpu, blackwell will offer "native" modern FP4 support and other bonus, but can be a bit more complicated setup.
24gb used mini is the cheapest (to buy and to run), but slowest, give you a full machine that can do other things.
a single 3090 still have more memory bandwith than 2 modern but 128 bits gpu, blackwell will offer "native" modern FP4 support and other bonus, but can be a bit more complicated setup.
24gb used mini is the cheapest (to buy and to run), but slowest, give you a full machine that can do other things.
sparks are nice for DEV in the cuda env, but lot of money for that very limited level of bandwith if you only interested in running things, depend quite a bit what you want to do.
One general tips, would be to try renting ont he cloud (5060, 3090, etc... are the type you can) to run your exact flow to see the actual difference
One general tips, would be to try renting ont he cloud (5060, 3090, etc... are the type you can) to run your exact flow to see the actual difference
- Joined
- May 18, 1997
- Messages
- 58,239
Not sure about that, but would be a nice setup for the memory footprint.
I run models on my 7900XTX (24GB), and it is super speedy. 16GB would be somewhat limiting.
For going 256GB, i would look Apple m3 ultra 256gb price vs 2x DGX spark...
The memory bandwith will be quite higher on apple here, ~800 GB/s, no FP4 support of course, Spark get more interesting if you know in advance you will use only nvpf4 models. and apparently now support have been added, but would verify to be sure.
Used Mac ultra 256GB seem in a similar price range, $8000 ish dollars.
The memory bandwith will be quite higher on apple here, ~800 GB/s, no FP4 support of course, Spark get more interesting if you know in advance you will use only nvpf4 models. and apparently now support have been added, but would verify to be sure.
Used Mac ultra 256GB seem in a similar price range, $8000 ish dollars.
Have you tried multiple GPUs running ollama or similar? From what I can find on the internet, 2x16GB will give you 32GB just like a single 32GB GPU so the only difference (hypothetically) between an RTX 5090 and 2x 5060 Ti 16GB is speed or token/s. What I want to know is if there's any hidden gotcha in doing this? PCI-E bandwidth becoming a problem? Anything like the old day with SLI where the rendering on both card not matching up and one would have to play with Alternate/Split Frame...
I'm not a fan of small and dumb models, even if they are fast. Deepseek v4 Flash 0731 NVFP4 with 1M context and 46 tok/s decode and some 1100 tok/s pre-fill just works.
Shame the pre-fill isn't much to write home about on Apple devices.
- Joined
- May 18, 1997
- Messages
- 58,239
No.
True enough, with that many * maybe that use case specific, better just:
https://aws.amazon.com/ec2/instance-types/mac/
If you can, actually try them before pulling the trigger.
I have no interest in any Apple products personally.
I have 4 Sparks on my desk and next week a MikroTik CRS504 will arrive so I finally can turn them into a 4-node cluster instead of the current 2-node clusters.
We shall see if 100G is fast enough for the interconnects, if not I will add one more switch at a later date to double the bandwidth.
I have 2x3090s running qwen 3.6 coder 30b. It's okay for pet projects and some minor code suggestions, but it's so much worse than something like fable or opus. It's fun but ultimately not that useful. If I get it to run openclaw or hermes and have it build stuff in unity over like 6 hours it can knock out a small pet project or feature if I just let it go. Some of it's useful, some of it is not.