What LLM/Model you using?

Jump on any major AI page and ask away at questions. I use Comet browser with perplexity for general free tier usage, about half my questions go to their built in chat vs basic web searches and I use their summarizer to summarize youtubes quickly before watching the video. If you want to tie an LLM into an agent that can do things with/for you, try something like Hermes Agent and a free tier model via Nous (which can added directly in settings>providers) to get you going.
 
antok86 said:
Where's a good starting point to lean what LLm is all about? I've got a bit of catching up to do.
Click to expand...
The nice thing about learning to use LLMs is you can just ask them. They know a lot about their own capabilities and limitations. I’m a few weeks into my journey of properly learning to use local models, having previously only dabbled, and I’ve surprised myself with how quickly I’m getting the hang of it. The biggest hurdle is just staring at that first chat console wondering “what do I even say to this thing?”

If you’re running local models on midrange gaming hardware (RX 9070 16GB, Ryzen 7700, 32 GB DDR5 in my case), then Strata and Unsloth Desktop will perform miracles on command. I’m not the best person to ask about the cloud-based AI that most people use, as I just don’t feel comfortable about handing over all my thoughts and data to the Zucks of the world.

So far, local LLMs have made a huge difference in my daily productivity. It’s good enough at summarizing web searches that I don’t bother to fight the algos anymore to get the info I need, I just ask something like “what is the community sentiment on places like Reddit and Github around [product x]? Do users feel it lives up to its marketing claims?” Then I would prompt it to put emphasis on comments that don’t use marketing speak, aren’t low-effort, don’t use similar words and phrases that the company’s website uses (the bot actually figured that one out on its own), etc. The output of a good model like Qwen 3.8 Flash-Next saves me a lot of time and mental effort, even when I don’t agree with its conclusions. You can ask it to summarize community sentiment around a product, criticisms of it, and figure out quickly what part of the praises and complaints applies to your use case and why because Qwen shows its work. If it’s a complex question or task, I just walk away and go do something else for a while. It’s already made a big difference in my life, and I’ve only just started.

I’m not much of a coder or hard-core software tinkerer, but I’ve managed to get it to write a script to automatically change my Mac’s resolution to 1080P when the TV connects via Sunshine + Artemis and then switch back to ultra-wide on disconnect. I’ve had it generate custom telemetry / ad / tracking blocklists for devices on my network that I don’t trust. I’ve had it figure out why software installs fail, or how to install tools that allow me to code and don’t phone home using Open WebUI. It’s not perfect, but it enables me to do things I wouldn’t have even tried before. My bigger “someday” projects feel very possible now, even if the jury’s out on whether the current tech and my efforts will be enough to get them to the finish line. I haven’t properly explored the reverse-engineering and emerging software-ownership possibilities yet, but I’m already planning how to get my old printer to print from my Mac and my phones, how to rip out the spyware from my smart TVs and devices, etc.

If I can do all of that with local models on modest hardware after a couple of weeks, then apply a multiplier for people who actually know what they are doing and can access frontier models or beefy hardware. I don’t want to get utopian (and there’s certainly a dystopian counter-argument), but I can easily see this technology creating a huge swell in entrepreneurship and cottage industries as motivated people rip and replace enshittified corporate moats with purpose-built tools. That vision may not be with us just yet (case in point, the sentiment around the Adobe clones from people who have actually tried to use them is dismal), but the genie is out of the bottle, and there’s no putting it back.
 
I have a powerful computer to test one out.

What would be good for it?

Specs: Intel 8700, 16GB RAM, and Nvidia Quadro P1000 4GB
 
IceDigger said:
I have a powerful computer to test one out.

What would be good for it?

Specs: Intel 8700, 16GB RAM, and Nvidia Quadro P1000 4GB
Click to expand...
Install Kolibri, then download Qwen 3.8 Max, expect one token a month or so.
 
IceDigger said:
Sweet!

I was given an NVidia Tesla M60 but it wont fit in the case :(
Click to expand...
Joking aside, not much at all will run on that system I'm afraid. With 4GB VRAM you can look at models with 1-3b parameters but they are so dumb it will be a waste of time unfortunately.
 
sphinx99 said:
I've been pretty impressed with qwen 3.8 flash next lately - it's quick and capable for its size.
Click to expand...
Saw some videos showing Strata + Qwen 3.8 Flash Next running decently on a 12G card but the catch is you need a lot of regular RAM to offload, minimum 64G.
 
I've been messing with Muse Glimmer 30B, QWEN 3.8 27B, qwopus3.8-27b-coder-flash (For coding recently, so far so good!), QWQ32B previously too. Mostly running on a 5090, or 2 x 3090s.

myzz said:
Saw some videos showing Strata + Qwen 3.8 Flash Next running decently on a 12G card but the catch is you need a lot of regular RAM to offload, minimum 64G.
Click to expand...
I'm excited to try this out when it releases as QWEN 4.
 
myzz said:
Saw some videos showing Strata + Qwen 3.8 Flash Next running decently on a 12G card but the catch is you need a lot of regular RAM to offload, minimum 64G.
Click to expand...
Can I have some ram please?

The Intel 8700 takes ddr4 2666 ram please. Only 2 slots.
 
Digital Viper-X- said:
I've been messing with Muse Glimmer 30B, QWEN 3.8 27B, qwopus3.8-27b-coder-flash (For coding recently, so far so good!), QWQ32B previously too. Mostly running on a 5090, or 2 x 3090s.


I'm excited to try this out when it releases as QWEN 4.
Click to expand...
Nice, how much system RAM do you have?
 
Digital Viper-X- said:
On the 5090 32, the dual 3090s 128 on epyc 8 channels populated.
Click to expand...
Nice, I've got the same! 4 channels though. Maybe I'll drop a card in the Epyc and see how it runs. Hilariously I got that 128G used on ebay for like 150$ a couple years ago :LOL: It was a steal even at the time.
 
I believe you can use a hidden command.. memmaker it is a magic command that gives you more RAMS!! You may need to look at old DOS releases to find it though.
 
If you are running qwen 3.8 with vllm 0.32.0 or earlier, move to 0.33.0 , considerable improvement on the token generation speed, went up from mid 60s on 2 x 3090s (27b 8bit) to 80ish for me (And I have them clock locked and power limited)

Quick benchmark to run(Change the URL and your model to match whatever you are running): (run it before and after upgrading)


Python: 
python3 - <<'PY'
import json
import time
import urllib.request

URL = "http://127.0.0.1:18081/v1/chat/completions"
MODEL = "lued/Qwen3.8-27B-INT8-W8A16-MTP"
MAX_TOKENS = 4096

payload = {
    "model": MODEL,
    "messages": [{
        "role": "user",
        "content": (
            "Write a very long, detailed technical explanation of database "
            "isolation levels, transaction anomalies, MVCC, locking, and "
            "distributed transactions. Include examples and continue until "
            "the output limit is reached."
        )
    }],
    "max_tokens": MAX_TOKENS,
    "temperature": 0.4,
    "top_p": 0.95,
    "stream": True,
    "stream_options": {"include_usage": True},
    "chat_template_kwargs": {
        "enable_thinking": True
    }
}

request = urllib.request.Request(
    URL,
    data=json.dumps(payload).encode("utf-8"),
    headers={"Content-Type": "application/json"},
    method="POST"
)

started = time.monotonic()
first_token = None
last_token = None
usage = {}
finish_reason = None
chunks = 0

print(f"Testing {MODEL}")
print(f"Requested completion: {MAX_TOKENS} tokens")
print("Generating...\n", flush=True)

with urllib.request.urlopen(request, timeout=300) as response:
    for raw in response:
        line = raw.decode("utf-8").strip()

        if not line.startswith("data: "):
            continue

        data = line[6:]

        if data == "[DONE]":
            break

        event = json.loads(data)

        if event.get("usage"):
            usage = event["usage"]

        if event.get("error"):
            raise RuntimeError(event["error"])

        for choice in event.get("choices", []):
            delta = choice.get("delta") or {}

            if any(delta.get(k) for k in (
                "content",
                "reasoning",
                "reasoning_content"
            )):
                now = time.monotonic()

                if first_token is None:
                    first_token = now

                last_token = now
                chunks += 1

            if choice.get("finish_reason"):
                finish_reason = choice["finish_reason"]

ended = time.monotonic()

tokens = usage.get("completion_tokens", 0)
prompt_tokens = usage.get("prompt_tokens", 0)

ttft = first_token - started if first_token else 0
decode_time = (
    last_token - first_token
    if first_token is not None and last_token is not None
    else 0
)

decode_tps = (
    (tokens - 1) / decode_time
    if decode_time > 0 and tokens > 1
    else 0
)

total_tps = tokens / (ended - started)

print("======================================")
print("        vLLM GENERATION BENCHMARK")
print("======================================")
print(f"Model:              {MODEL}")
print(f"Completion tokens:  {tokens}")
print(f"Prompt tokens:      {prompt_tokens}")
print(f"Streaming chunks:   {chunks}")
print(f"Time to first:      {ttft:.2f}s")
print(f"Decode duration:    {decode_time:.2f}s")
print(f"Total duration:     {ended-started:.2f}s")
print("--------------------------------------")
print(f"Decode throughput:  {decode_tps:.2f} tok/s")
print(f"End-to-end speed:   {total_tps:.2f} tok/s")
print(f"Finish reason:      {finish_reason}")
print("======================================")
PY
 
Last edited:
  • Like
Reactions: myzz
like this
Hey everyone, I'm looking for advise on the best local LLM setup for autonomous coding on a 4070 Ti Super with 16GB
I'm building a local autonomous development system and would appreciate some input from people running similar setups.

Current hardware:
  • RTX 4070 Ti SUPER, 16GB VRAM
  • 64GB system RAM
  • Windows with Ubuntu WSL2
  • Secondary Mac mini with 16GB RAM
  • Third MacBook Pro M1Max 32GB Ram
  • Fourth RTX 2080 Ti Super, 11GB VRAM
Current stack:
  • Ollama with Qwen3.5 9B (Q4_K_M)
  • OpenCode
  • Python-based task orchestration
  • Git, Flutter/Dart, Java, and other development work

What I'm trying to accomplish:

I want an autonomous coding agent that can take approved Jira backlog items, inspect the codebase, implement changes, write tests, run validation, diagnose failures, and retry when necessary.
Ideally, I review the verified results and decide what gets merged.
I've already built a working automated verification pipeline using Codex for the heavier coding tasks.
Now I'm trying to determine how much of that workload I can reliably move onto local hardware.


My questions:
  1. Is Qwen3.5 9B a sensible choice for this hardware, or would a larger quantized coding model deliver better results?
  2. Are DeepSeek, Qwen-Coder, or other models noticeably better for multi-file coding and agentic tool use?
  3. Would you recommend a single model or separate coding and review models?
  4. Are there better orchestration tools or model/server configurations anyone can share?
  5. Has anyone achieved reliable unattended coding with automated testing and repair on similar hardware? If not what did it take to enable this workflow?
I'm prioritizing correctness, successful task completion, and throughput, not just tokens per second.
I don't mind longer inference times if the model actually completes more work with fewer retries.
Any firsthand experience or benchmark suggestions would be appreciated!
Thanks in advance!
 
I was playing around with qwen3.8:27b on my 3090 and it runs really good but I was having some issues with it un the webui / ollama where it was stopping at "thinking" a lot of the time and supposedly this was an issue there were working on. I am using it to plot out my retirement to a homestead plan to brainstorm and flesh out the plan.
 
motqalden said:
I was playing around with qwen3.8:27b on my 3090 and it runs really good but I was having some issues with it un the webui / ollama where it was stopping at "thinking" a lot of the time and supposedly this was an issue there were working on. I am using it to plot out my retirement to a homestead plan to brainstorm and flesh out the plan.
Click to expand...
Qwen 3.8 27B and other variants have an issue where they will get stuck in a loop auditing their own thoughts for a while, if you have their reasoning effort on xhigh. I've seen it happen multiple times on coding tasks and had to switch to a different agent / plugin that helps break it out of loops.
 
After a couple months of using cloud models vs local on Mac Studio 256GB, I prefer running on the cloud (/duck). For $20/mo I can run mid tier models like DeepSeek 4.1 Flash all day long (for my usage) with Kimi-k3.1-pro for deep dives, mistral-large-4, inkling-2-small for ocr, and others for some tasks. DS4.1Flash is daily driver and I have it run some deep dive tasks comparisons with other models from time to time to find which demonstratively provides better results for a given task. Even with with some handoffs to more expensive models like Kimi I easily stay under $20 a month usage. My Mac Studio will prob be on sale soon.

I do get it though, some data just cannot go online. My employer has a strict policy on that. There are cloud providers that explicitly state they do not train on your data if that is your general worry. I'm not a coder and I'm not doing work on AI.

I do find it to be a very good personal assistant and system admin for my personal system and have no reservation of using a cloud models for that.
 
FSCDiablo said:
After a couple months of using cloud models vs local on Mac Studio 256GB, I prefer running on the cloud (/duck). For $20/mo I can run mid tier models like DeepSeek 4.1 Flash all day long (for my usage) with Kimi-k3.1-pro for deep dives, mistral-large-4, inkling-2-small for ocr, and others for some tasks. DS4.1Flash is daily driver and I have it run some deep dive tasks comparisons with other models from time to time to find which demonstratively provides better results for a given task. Even with with some handoffs to more expensive models like Kimi I easily stay under $20 a month usage. My Mac Studio will prob be on sale soon.

I do get it though, some data just cannot go online. My employer has a strict policy on that. There are cloud providers that explicitly state they do not train on your data if that is your general worry. I'm not a coder and I'm not doing work on AI.

I do find it to be a very good personal assistant and system admin for my personal system and have no reservation of using a cloud models for that.
Click to expand...
Yup. Been here a while. All fun to experiment and run local but end of day this is the easy and economical path.
 
You must log in or register to reply here.
Back
Top