What LLM/Model you using?

[FrgMstr]

Figure this would be a good thread to share information on what you are using locally for AI. I am running the Strix 395+ 128GB machine as my "Home AI Server." Using LM Studio on my desktop to connect to the server over the network using LM Link. I am using Hermes Desktop exclusively now, and finally really understanding it and how to get it dialed in for specific profiles/tasks.

I have used a lot of variants of Qwen and Gemini lately, but started using unsloth's laguna-s-2.1@q4_k_s in the last couple of days. I have having MUCH better workflow results with this Laguna than I was with Qwen/Gemini/etc. Did have some issues getting it to fit in the memory footprint, along with microsoft/phi-4-mini-reasoning as my Aux LLM for specific tasks.

Laguna S 2.1 is a 118B total parameter Mixture-of-Experts model with 8B activated parameters per token, designed for agentic coding and long-horizon work.

 

[Supercharged_Z06]

Also have a Strix 395+ 128GB machine (GMKtec Evo 2), but been too busy with home automation shenanigans and getting everything setup infrastructure wise with a new house build to be able to really dive in and mess around with it yet.

So, can't really contribute anything just yet, but looking forward to gleaning some good ideas/info off this thread.

 

[cdabc123]

I have a mi50 that can run 8b and 14b models. I havent really got it good enough for a coding agent. Have been using codex 20$ and have been making some pretty good progress in python and fpga development.

 

[FrgMstr]

Also have a Strix 395+ 128GB machine (GMKtec Evo 2), but been too busy with home automation shenanigans and getting everything setup infrastructure wise with a new house build to be able to really dive in and mess around with it yet.

So, can't really contribute anything just yet, but looking forward to gleaning some good ideas/info off this thread.

You using Hermes?

 

[Supercharged_Z06]

You using Hermes?

Not yet - been just manually brute forcing YAML within Home Assistant (with very little AI help) on some pretty basic automations - mostly just modifying/tinkering with stuff to get basic automations up and running.

Want to eventually build out an AI agent to assist with more complex logic focused on automation coding tasks within Home Assistant. Have read that lots of folks have really liked using Claude Code locally to assist in HA automation.

 

[FrgMstr]

Not yet - been just manually brute forcing YAML within Home Assistant (with very little AI help) on some pretty basic automations - mostly just modifying/tinkering with stuff to get basic automations up and running.

Want to eventually build out an AI agent to assist with more complex logic focused on automation coding tasks within Home Assistant. Have read that lots of folks have really liked using Claude Code locally to assist in HA automation.

Hermes has full skills built for Home Assistant. Don't fight it, use your hardware.

https://hermes-agent.nousresearch.com/docs/user-guide/messaging/homeassistant

 

[FrgMstr]

Not yet - been just manually brute forcing YAML within Home Assistant (with very little AI help) on some pretty basic automations - mostly just modifying/tinkering with stuff to get basic automations up and running.

Want to eventually build out an AI agent to assist with more complex logic focused on automation coding tasks within Home Assistant. Have read that lots of folks have really liked using Claude Code locally to assist in HA automation.

Worth noting on moving forward with Hermes. It is extremely important to build a profile that is specific to the tasks you are trying to do and give it the needed skills and tools to be successful.

 

[Supercharged_Z06]

Hermes has full skills built for Home Assistant. Don't fight it, use your hardware.

https://hermes-agent.nousresearch.com/docs/user-guide/messaging/homeassistant

See, that's exactly why I said I'd be following this thread.

 

[FrgMstr]

See, that's exactly why I said I'd be following this thread.

My default profile/agent I built now manages all agents on the machine, and babysits the overall health of all systems, and it researches and evaluates on how to build other agents to do specific jobs then builds those agents.

Soul.md: "You are Hermes Agent, created by Nous Research. Your purpose is specifically managing and maintaining the Hermes agent framework itself.

Your scope includes: setting up and configuring profiles; creating, editing, and reviewing system prompts (SOUL.md files); organizing skills, tools, cron jobs, and memory across profiles; troubleshooting Hermes processes and infrastructure; and handling meta-level configuration tasks. This is your sole focus - you are the "meta-agent" that keeps Hermes running smoothly.

You can handle general user project work if a specialized profile is not available, but it is not your focus (coding projects, creative writing, research assistance, etc.). Those tasks belong in specialized profiles created for specific purposes. When a request falls outside of direct Hermes management/maintenance, either route it to an appropriate profile or create one if needed.

Communicate clearly and directly. Admit uncertainty when appropriate. Prioritize being genuinely useful over verbose responses unless asked otherwise."

 

[FrgMstr]

I have a mi50 that can run 8b and 14b models. I havent really got it good enough for a coding agent. Have been using codex 20$ and have been making some pretty good progress in python and fpga development.

Codex really shines IMO for $20. You can get a LOT out of that.

 

[StoleMyOwnCar]

I have a DGX Spark and I primarily run Qwen 3.5 122B int 4 autoround on it (well it's a hybrid int4/q8 quant). Gets about 50 tok/s gen on average at 260k context size. Have had some issues with looping, but otherwise it's a pretty smart model, and image interpretation capable. I used to use a much harder to set up version of it, but this repo sets it up without issues:
https://github.com/Entrpi/qwen3.5-122B-A10B-on-spark

There's also an uncensored version of that, which I have used. It's just a drop-in replacement with minimal modifications to the start bash script: https://huggingface.co/happypatrick/Qwen3.5-122B-A10B-heretic-int4-AutoRound


I have also used some forks of antirez's Deepseek 4 Flash engine, like this one
https://github.com/Entrpi/ds4-on-spark
Gets about 30 tok/s and 1k tok/s prefill. A lot of work and love went into trying to optimize DSV4F on the DGX spark, major props to the community.

I have also used Qwen 3.6 35B A3B on it, which gets about 150 tok/s at NVFP4, but is a bit stupid.

For my 5090, I've run 27B on it normally, and it was a very capable model.

I tried Laguna 2.1, but the NVFP4 quant of it is still messed up. I don't think I've ever gotten good results out of it. Am considering another quantization.

There are a few other models that I'm considering on my Spark, but so far Qwen 3.5 122B is kind of king. It's overall the most intelligent I have used. DSV4F is okay, but this latest 731 update has behaved oddly on the lower quants. It doesn't feel as intelligent. They might fix that. DSV4F runs very well on two sparks if you connect them with a cable, but unfortunately I can't find a good deal on another spark. I got this one for 3k, before prices just went crazy.

Still haven't tried any agentic stuff, open to suggestions on how to start that.

 

[FrgMstr]

I pretty much exclusively used Hermes CLI through WSL/Ubuntu till a couple of weeks ago. The Hermes Desktop (Windows) is friggin great. You still have to get to bash/powershell to get things done once in a while, but the Desktop has gotten amazing since first launch. Set your default agent/profile Soul.md file with what I pasted above and it will get you way down the road to using Hermes' abilities. laguna-s-2.1@q4_k_s has made it all shine though. It feels like what I thought agents should be when we first started playing with this months ago. I have not seen a failed reasoning loop that required intervention. It did finally break this morning, not sure why, just unloaded. And it is pretty darn speedy as well.

 

[LukeTbk]

gemini and GPT 5.6 are the 2 I use the most right now, the GPT 5.6 monthly deal is really hard to beat value wise.

 

[FrgMstr]

gemini and GPT 5.6 are the 2 I use the most right now, the GPT 5.6 monthly deal is really hard to beat value wise.

Been using Claude Opus 5.0. I generally burn through 50% of my session allotment in one prompt. Moving over to Codex to give it a try. I have SuperGrok and I am not even using Build. I am halfway convinced it is better than Opus 5.0.

 

[Shoganai]

Mostly MiMo 2.5 (as cheap as DeepSeek but was designed from the ground up to work inside agentic harnesses ... DeepSeek fails hard in hardnesses) and for more advanced tasks and auditing my final projects, Kimi K3, Claude Opus / Fable, and GPT 5.6. Also Gemini with Antigravity for specific things. I run everything through Hermes via Telegram and Terminal and occasionally the dedicated desktop app.

 

[4saken]

Been using Claude Opus 5.0. I generally burn through 50% of my session allotment in one prompt. Moving over to Codex to give it a try. I have SuperGrok and I am not even using Build. I am halfway convinced it is better than Opus 5.0.

I've pretty much abandoned everything else and moved to Grok 4.5 and using build heavily. Finally feel like it is close enough where I don't need to keep shopping and every iteration has been massively significant as of late. Still using 5090 locally for hermes/long running loops etc with 27B, but have all of my remote agents configured to use Grok. Credits feel pretty generous compared to every other plan ive tried and the weekly reset is convenient. And tbh, just have new model fatigue across the board.

 

[FrgMstr]

new model fatigue across the board

Yeah, between just staying up to date with what is launching is enough to drive you nuts. Then figuring out if it works for you...whole other enchilada.

I doubt we will see a Laguna 2.1 quant that will fit in 5090's VRAM, but this model is a game changer for agentic and coding. Its reasoning is a few levels above Qwen 3.6 27B/35B.

 

[pandather]

I just got my 5090 and have been a teeny bit overwhelmed with the models. Tried probably 20 of them. Mainly for programming (generating code), but also a lot for generating structured outputs for my programs. As for the ones that fit I like qwen 3.6 the best I think, followed by gemma 4 if I need vision. I started messing with the Minimax 2.7 MoE model and the 3 bit XXS quant barely fits on my system with 96GB of ram while allowing me to load larger files and context, but at 20 tok/s it’s quite usable and has been generating much better results than the other models.

 

[sphinx99]

Figure this would be a good thread to share information on what you are using locally for AI. I am running the Strix 395+ 128GB machine as my "Home AI Server." Using LM Studio on my desktop to connect to the server over the network using LM Link. I am using Hermes Desktop exclusively now, and finally really understanding it and how to get it dialed in for specific profiles/tasks.

I have used a lot of variants of Qwen and Gemini lately, but started using unsloth's laguna-s-2.1@q4_k_s in the last couple of days. I have having MUCH better workflow results with this Laguna than I was with Qwen/Gemini/etc. Did have some issues getting it to fit in the memory footprint, along with microsoft/phi-4-mini-reasoning as my Aux LLM for specific tasks.

Laguna S 2.1 is a 118B total parameter Mixture-of-Experts model with 8B activated parameters per token, designed for agentic coding and long-horizon work.

View attachment 818135
View attachment 818136

I'm lucky to have a pair of M3 Ultra studios with 256GB and a single one with 512GB, chained. I've been able to run Deepseek V4 Pro at 4-bit quant but there are some rather severe context window limitations. Llama 3.1 405B at 16-bit runs fairly well and is probably the best / most useful model I can run locally. I wish I had gotten more 512GB Studios when they were available, I'd be able to play with K3 @ 4bit which would be fun But, this setup is for overnight desktop characterization of work models while @ home, so all of this is just kick-the-tires stuff.

 

[FrgMstr]

I'm lucky to have a pair of M3 Ultra studios with 256GB and a single one with 512GB, chained. I've been able to run Deepseek V4 Pro at 4-bit quant but there are some rather severe context window limitations. Llama 3.1 405B at 16-bit runs fairly well and is probably the best / most useful model I can run locally. I wish I had gotten more 512GB Studios when they were available, I'd be able to play with K3 @ 4bit which would be fun But, this setup is for overnight desktop characterization of work models while @ home, so all of this is just kick-the-tires stuff.

sphinx99 over there just big-ballin'!

 

[mashie]

Figure this would be a good thread to share information on what you are using locally for AI. I am running the Strix 395+ 128GB machine as my "Home AI Server." Using LM Studio on my desktop to connect to the server over the network using LM Link. I am using Hermes Desktop exclusively now, and finally really understanding it and how to get it dialed in for specific profiles/tasks.

I have 4x DGX Sparks for my home AI needs. Currently configured as two dual clusters, both running Deepseek-v4-Flash-0731.

Once they are in a single 4-node cluster they will get GLM-5.2 on there.

 

[cjcox]

I'm using the LLM that respects your privacy and isn't all about exploiting you and others. That is, I'm not using one.

 

[applegrcoug]

I have 4x DGX Sparks for my home AI needs. Currently configured as two dual clusters, both running Deepseek-v4-Flash-0731.

Once they are in a single 4-node cluster they will get GLM-5.2 on there.

You may want to try glm 5.2 with colibri....might work. I want to try it out but I'm not sure I have the ram to do it efficiently.

I've been tinkering with the v4 flash the past few days and it was able to chug through some stuff I was impressed it could. It seems to be handling stuff better than minimax m2.7.

Apparently next week qwen is dropping 3.8 27b, that will be fun.

Months ago, I set up openclaw and got it doing one task pretty well; scraping several different point weather forecasts and sending out a daily summary. It worked alright, I set it up using gpt 5.5 and then minimax m2.7 was supposed to run the job. Problem was, often times, it ended up being gpt because the delegation broke or something.

Right now, I'm in the process of setting up hermes. I just finished porting over the weather summary. I used gpt 5.6 sol to rebuild it, but now I'm just having a smaller gemini model run it and seems to be working well enough after a day. This time with hermes, I'm hoping to accomplish more and work on a system to work with water management better. I plan to use several profiles to try and get this one figured out. gpt will be part of it, but I can see my tokens getting burned quickly, so I want it to act more as a reviewer.

My system is a bit unconventional and lately been thinking I should liquidate it and get a couple sparks and cluster them...I have a 6x3090 and am tapped out with what I can do with it.


......looking at ebay at 3090's, Christ, when did they get worth that?!

 

[StoleMyOwnCar]

Once they are in a single 4-node cluster they will get GLM-5.2 on there.

Let me know how that is, vs DSV4F. I've seen people talking about running GLM, wondering about what the quality is like vs the best 2 spark solution. Obviously 4 nodes you're talking like 16k though so I hope you have some ideas for ROI on that.

That's what I meant when I thought that the Spark was currently kind of the place to be, though. At least you have incremental, linear upgrade paths, and they actually do scale well like that. With any other platform, you're basically shelling out way more and ditching the whole thing if you want to upgrade.

I'm using the LLM that respects your privacy and isn't all about exploiting you and others. That is, I'm not using one.

Local LLMs literally have no internet connectivity unless you allow it. You have 100% privacy.

Unless you mean the whole system around AI in general and how it was basically created via pirating and systematic exploitation in some ways, then, sure. Unfortunately that already happened. You can't put the genie back in the bottle.

Personally I don't want to pay for any of these fucking subscription services because I hate all this datacenter shit. Even if they're technically a better deal than me running locally, I want my own privacy and I don't want more incentives to build more datacenters by feeding into that. So I'll just use free chatgpt and shit. If I can siphon money from them, I will.

Unfortunately, Nvidia is still getting the money when you buy any of this though, except I guess used 3090s.