• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

What LLM/Model you using?

FrgMstr

FrgMstr

Just Plain Mean
Staff member
2FA
Joined
May 18, 1997
Messages
58,197
Figure this would be a good thread to share information on what you are using locally for AI. I am running the Strix 395+ 128GB machine as my "Home AI Server." Using LM Studio on my desktop to connect to the server over the network using LM Link. I am using Hermes Desktop exclusively now, and finally really understanding it and how to get it dialed in for specific profiles/tasks.

I have used a lot of variants of Qwen and Gemini lately, but started using unsloth's laguna-s-2.1@q4_k_s in the last couple of days. I have having MUCH better workflow results with this Laguna than I was with Qwen/Gemini/etc. Did have some issues getting it to fit in the memory footprint, along with microsoft/phi-4-mini-reasoning as my Aux LLM for specific tasks.

Laguna S 2.1 is a 118B total parameter Mixture-of-Experts model with 8B activated parameters per token, designed for agentic coding and long-horizon work.

1785620176391.png

1785620215614.png
 
Also have a Strix 395+ 128GB machine (GMKtec Evo 2), but been too busy with home automation shenanigans and getting everything setup infrastructure wise with a new house build to be able to really dive in and mess around with it yet.

So, can't really contribute anything just yet, but looking forward to gleaning some good ideas/info off this thread. :D
 
I have a mi50 that can run 8b and 14b models. I havent really got it good enough for a coding agent. Have been using codex 20$ and have been making some pretty good progress in python and fpga development.
 
Supercharged_Z06 said:
Also have a Strix 395+ 128GB machine (GMKtec Evo 2), but been too busy with home automation shenanigans and getting everything setup infrastructure wise with a new house build to be able to really dive in and mess around with it yet.

So, can't really contribute anything just yet, but looking forward to gleaning some good ideas/info off this thread. :D
Click to expand...
You using Hermes?
 
FrgMstr said:
You using Hermes?
Click to expand...
Not yet - been just manually brute forcing YAML within Home Assistant (with very little AI help) on some pretty basic automations - mostly just modifying/tinkering with stuff to get basic automations up and running.

Want to eventually build out an AI agent to assist with more complex logic focused on automation coding tasks within Home Assistant. Have read that lots of folks have really liked using Claude Code locally to assist in HA automation.
 
Supercharged_Z06 said:
Not yet - been just manually brute forcing YAML within Home Assistant (with very little AI help) on some pretty basic automations - mostly just modifying/tinkering with stuff to get basic automations up and running.

Want to eventually build out an AI agent to assist with more complex logic focused on automation coding tasks within Home Assistant. Have read that lots of folks have really liked using Claude Code locally to assist in HA automation.
Click to expand...
Hermes has full skills built for Home Assistant. Don't fight it, use your hardware. :)

https://hermes-agent.nousresearch.com/docs/user-guide/messaging/homeassistant
 
Supercharged_Z06 said:
Not yet - been just manually brute forcing YAML within Home Assistant (with very little AI help) on some pretty basic automations - mostly just modifying/tinkering with stuff to get basic automations up and running.

Want to eventually build out an AI agent to assist with more complex logic focused on automation coding tasks within Home Assistant. Have read that lots of folks have really liked using Claude Code locally to assist in HA automation.
Click to expand...
Worth noting on moving forward with Hermes. It is extremely important to build a profile that is specific to the tasks you are trying to do and give it the needed skills and tools to be successful.
 
Supercharged_Z06 said:
See, that's exactly why I said I'd be following this thread. :p
Click to expand...
My default profile/agent I built now manages all agents on the machine, and babysits the overall health of all systems, and it researches and evaluates on how to build other agents to do specific jobs then builds those agents.

Soul.md: "You are Hermes Agent, created by Nous Research. Your purpose is specifically managing and maintaining the Hermes agent framework itself.

Your scope includes: setting up and configuring profiles; creating, editing, and reviewing system prompts (SOUL.md files); organizing skills, tools, cron jobs, and memory across profiles; troubleshooting Hermes processes and infrastructure; and handling meta-level configuration tasks. This is your sole focus - you are the "meta-agent" that keeps Hermes running smoothly.

You can handle general user project work if a specialized profile is not available, but it is not your focus (coding projects, creative writing, research assistance, etc.). Those tasks belong in specialized profiles created for specific purposes. When a request falls outside of direct Hermes management/maintenance, either route it to an appropriate profile or create one if needed.

Communicate clearly and directly. Admit uncertainty when appropriate. Prioritize being genuinely useful over verbose responses unless asked otherwise."
 
Last edited:
cdabc123 said:
I have a mi50 that can run 8b and 14b models. I havent really got it good enough for a coding agent. Have been using codex 20$ and have been making some pretty good progress in python and fpga development.
Click to expand...
Codex really shines IMO for $20. You can get a LOT out of that.
 
I have a DGX Spark and I primarily run Qwen 3.5 122B int 4 autoround on it (well it's a hybrid int4/q8 quant). Gets about 50 tok/s gen on average at 260k context size. Have had some issues with looping, but otherwise it's a pretty smart model, and image interpretation capable. I used to use a much harder to set up version of it, but this repo sets it up without issues:
https://github.com/Entrpi/qwen3.5-122B-A10B-on-spark

There's also an uncensored version of that, which I have used. It's just a drop-in replacement with minimal modifications to the start bash script: https://huggingface.co/happypatrick/Qwen3.5-122B-A10B-heretic-int4-AutoRound


I have also used some forks of antirez's Deepseek 4 Flash engine, like this one
https://github.com/Entrpi/ds4-on-spark
Gets about 30 tok/s and 1k tok/s prefill. A lot of work and love went into trying to optimize DSV4F on the DGX spark, major props to the community.

I have also used Qwen 3.6 35B A3B on it, which gets about 150 tok/s at NVFP4, but is a bit stupid.

For my 5090, I've run 27B on it normally, and it was a very capable model.

I tried Laguna 2.1, but the NVFP4 quant of it is still messed up. I don't think I've ever gotten good results out of it. Am considering another quantization.

There are a few other models that I'm considering on my Spark, but so far Qwen 3.5 122B is kind of king. It's overall the most intelligent I have used. DSV4F is okay, but this latest 731 update has behaved oddly on the lower quants. It doesn't feel as intelligent. They might fix that. DSV4F runs very well on two sparks if you connect them with a cable, but unfortunately I can't find a good deal on another spark. I got this one for 3k, before prices just went crazy.

Still haven't tried any agentic stuff, open to suggestions on how to start that.
 
You must log in or register to reply here.
Back
Top