What LLM/Model you using?

[M76]

It can vary just giving it the same task, repeatedly. Downside of probabilistic vs determinsistic.

It's like calling an indian call center, hoping for the most intelligent least arrogant operator, but often that's not what you get.

 

[mnewxcv]

Qwen 3.5 122b moe replaced gpt oss 120b for me. I don't know if it's more accurate, but I like it's output better and I like that it has vision. Also, it has huge context which I like because most of the conversations I have are iterative. If I need a quick answer, I usually throw it into google/Gemini, claude, or chatgpt, depending on what it is. Google ai has provided tons of bad into that I call it out on though for technical things.

It will be nice when windows updates how it uses unified memory to allow more to be used by the gpu, later this year.

Also looking to start using medgemma to track things I want to bring up with my Dr between physicals. I seem to always forget things like my shoulder issue when it's fine the week of the appointment, etc. But it might be cool to see if ai has any insights and medgemma seems solid. ***only local of course, don't be discussing your personal health with cloud ai***

 

[4saken]

Curious if folks complaining about all local models being dumb actually tune the parameters of their model. It is not insignificant by a long stretch.

 

[cdabc123]

Most models are simply dumb. They start to resemble intelligence around 120b or so, where maybe if you use them right the can be usefull. Anything under 8b is ofc useless for almost all things.

The problem is, to do tasks like advanced coding, you dont need a model that is just good enough, you need pretty much the best model avalible and even then you must prompt it and check work and use a good workflow. A cloud model is like 1.7t there really is no comparison between that and a local model.

It is still cool to see the usefulness many can get with over 32gb of vram or fast mem. The sparks and macs and good gpus are definitely capable of some work.

 

[mnewxcv]

Most models are simply dumb. They start to resemble intelligence around 120b or so, where maybe if younusebthem right the can be usefull. Anything under 8b is ofc useless for almost all things.

The problem is, to do tasks like advanced coding, you dont need a model that is just good enough, you need pretty much the best model avalible and even then you must prompt it and check work and use a good workflow. A cloud model is like 1.7t there really is no comparison between that and a local model.

It is still cool to see the usefulness many can get with over 32gb of vram or fast mem. The sparks and macs and good gpus are definitely capable of some work.

The small models can still be good for tasks instead of knowledge. And if they're able to search the web, they can be pretty useful.

 

[LukeTbk]

With tooling and agentic harness small model can be quite useful and with 512GB Mac connected together and a lot of money you can run large model, the distinction is a bit more cheap to run vs expensive than local vs non local.

 

[SecretStash]

"Advanced Coding" can only be advanced with developer guidance. Developers would not be needed if there's true advanced coding but we aren't there yet. If we get there then there will be no more software development careers. But a good developer knows how to solve very advanced problems by using the AI skillfully on focused prompts.

 

[4saken]

With tooling and agentic harness small model can be quite useful and with 512GB Mac connected together and a lot of money you can run large model, the distinction is a bit more cheap to run vs expensive than local vs non local.

100% Think a harness alone would solve many of the "My model is dumb" experiences. Don't want to say people are doing it wrong, but I don't have these issues from 2.5b up to 35b locally and being properly utilized for the intended tasks.

 

[MrGuvernment]

"Advanced Coding" can only be advanced with developer guidance. Developers would not be needed if there's true advanced coding but we aren't there yet. If we get there then there will be no more software development careers. But a good developer knows how to solve very advanced problems by using the AI skillfully on focused prompts.

Goes back to using the right tool for the job, and knowing how to properly use said tool to be a force multiplier, vs just a crutch for lack of actual knowledge and skill.

 

[applegrcoug]

I have no interest in Colibri, but you can just about fit GLM-5.2 on a cluster of 4 on a 4bit quant.

Hermes + DSv4 Flash 0731 is a match made in heaven. Last night I asked it to figure out why a MySQL database in my system always errored out when trying to make a backup. Well the issue was found, a backup was made, a MariaDB was installed on a Raspberry Pi to verify it could be restored. Then after a pro/con discussion about MySQL vs MariaDB it was decided to migrate the database to MariaDB as it was under support. Oh and now I have a monthly scheduled backup in place. All in all about 40 minutes and no external LLM used.

Colour me impressed.

Well, I tried it...0.3tok/sec.

I too have been using the deepseek 0731. I've been doing the q3, but considering the way the made the model, i do t think that is too bad. Also at the full 1m context. Only now getting it cranking on some stuff.

 

[mashie]

Well, I tried it...0.3tok/sec.

I too have been using the deepseek 0731. I've been doing the q3, but considering the way the made the model, i do t think that is too bad. Also at the full 1m context. Only now getting it cranking on some stuff.

I'm running the NVFP4 version with FP8 KV Cache. The 1M context is stable as a 950K token needle in the haystack test worked fine. Around 1100 token/s pre-fill and around 45 token/s decode on single stream.

 

[Dan_D]

I generally use Qwen and Flux 1/Flux 2 via Pinokio (provides a UI) for a lot of my needs. (Mostly image generation and editing.)

 

[Zarathustra[H]]

I've made no secret of the fact that I am not a huge fan of our modern AI era, but at the same time my biggest (but not only) complaint about it has been the privacy implications of using frontier models. I figured I was doing myself a disservice by not trying to learn as much as possible about it, so for the last several months I have been trialing several different local inference (since they don't leak anything outside my local machine) models in Ollama with an OpenWebUI fontend on the CPU on my EPYC 7543 server. It is slow. But not quite as slow as it would be on consumer hardware, as the Milan EPYC system has 8x DDR4-3200 memory channels, giving me quite a lot of memory bandwidth.

My response output from a model like Llama 3.3 70B (Q4_K_M) has been about 3.3 tokens/s, which is... ....tolerable. If only barely.

The real killer when running on the CPU is pre-fill, especially when running a frontend like OpenWebUI as it inserts a lot of stuff into the prompt that is sent to Ollama. CPU's are nowhere near as fast at the type of matrix operations required for pre-fill that highly parallelized GPU's are.

Either way, the performance here was secondary. I wanted to evaluate the capability of a 70B class model (the largest I can reasonably expect to run on any hardware available at a price I'd be willing to pay. So I've been plinking away at it. Typing a prompt, hitting submit, and going to get a cup of coffee...

Well, my evaluation period has come to an end. I decided to take the plunge and I have a 64GB AMD Instinct MI210 card incoming.

I have been doing a lot of going back and forth regarding what models to run on it given the 64GB VRAM limitation.

My use case will be mostly web-assisted knowledge gathering/research.

The MI210 does - unfortunately - not get me enough VRAM to run any of the exciting new MoE models, but it does allow me to tear through some of the traditional dense 70-75B models out there. (any hardware with enough VRAM to run the likes of Llama 4 Scout was just WAY out of budget)

Firstly, since the model will be used for information gathering and research I decided to avoid any of the Chinese models, as I suspect censorship and bias impact their training.

This led me to rather quickly settle on Llama 3.3 70B (Q5_K_M) as the best capability I could fit in the 64GB I have at my disposal. This ought to use about 50GB of VRAM.

In addition to this I plan on running a smaller model that uses less VRAM assigned as a task model to offload that from the main heavy model. I haven't decided which one yet though.

One good choice is Llama 3.2 3B which ought to be lightning fast on those task threads that don't require the reasoning or knowledge of a heavier model, while also using very little VRAM (~2GB) and thus not cutting down to my context window so much. This would leave ~14GB for the context window which gets me about 42K tokens which is quite respectable for what I plan to do with it.

Another option is to go with a secondary task model that is a little bit larger but also has vision capability that can be triggered when the web assist digests webpages with images. I initially considered Llama 3.2 11B Vision, but at ~7.5GB for the weights it is quite large and really would hamper my context window size for the main model.

Since I'll be using this model for both tasks and as a companion model for image processing, I am still eliminating Chinese models, as I don't want any censorship/bias in the creation of web search terms for the web assist (which is one of the things the task model does) There are a dizzying array of options here, Gemma 4 4B, Gemma 3 4B, Phi-4-Multimodal and many many more.

One of the enormous differences I expect to see is that the MI210 will absolutely rip through prompt evaluation / prefill at 500-1200 tokens/s, due to GPU's higher matrix math capability meaning the actual evaluation phase where the model is providing its response will start almost instantaneously. Once the output phase starts I expect to see between 25 and 30 tokens per second out of Llama 3.3 70B (Q5_K_M) which should be more than sufficient for my desired purposes.

All of this is - however - theoretical at this point. Once I actually get the MI210 set up and start playing with it, I'm sure large parts of this plan will change. Nothing beats actual hands on experience. Once I can actually execute prompts at a reasonable speed, I'm sure the hands on lessons will go from a trickle to something more closely resembling a blast from a fire hose.

 

[Zarathustra[H]]

I've made no secret of the fact that I am not a huge fan of our modern AI era, but at the same time my biggest (but not only) complaint about it has been the privacy implications of using frontier models. I figured I was doing myself a disservice by not trying to learn as much as possible about it, so for the last several months I have been trialing several different local inference (since they don't leak anything outside my local machine) models in Ollama with an OpenWebUI fontend on the CPU on my EPYC 7543 server. It is slow. But not quite as slow as it would be on consumer hardware, as the Milan EPYC system has 8x DDR4-3200 memory channels, giving me quite a lot of memory bandwidth.

My response output from a model like Llama 3.3 70B (Q4_K_M) has been about 3.3 tokens/s, which is... ....tolerable. If only barely.

The real killer when running on the CPU is pre-fill, especially when running a frontend like OpenWebUI as it inserts a lot of stuff into the prompt that is sent to Ollama. CPU's are nowhere near as fast at the type of matrix operations required for pre-fill that highly parallelized GPU's are.

Either way, the performance here was secondary. I wanted to evaluate the capability of a 70B class model (the largest I can reasonably expect to run on any hardware available at a price I'd be willing to pay. So I've been plinking away at it. Typing a prompt, hitting submit, and going to get a cup of coffee...

Well, my evaluation period has come to an end. I decided to take the plunge and I have a 64GB AMD Instinct MI210 card incoming.

I have been doing a lot of going back and forth regarding what models to run on it given the 64GB VRAM limitation.

My use case will be mostly web-assisted knowledge gathering/research.

The MI210 does - unfortunately - not get me enough VRAM to run any of the exciting new MoE models, but it does allow me to tear through some of the traditional dense 70-75B models out there. (any hardware with enough VRAM to run the likes of Llama 4 Scout was just WAY out of budget)

Firstly, since the model will be used for information gathering and research I decided to avoid any of the Chinese models, as I suspect censorship and bias impact their training.

This led me to rather quickly settle on Llama 3.3 70B (Q5_K_M) as the best capability I could fit in the 64GB I have at my disposal. This ought to use about 50GB of VRAM.

In addition to this I plan on running a smaller model that uses less VRAM assigned as a task model to offload that from the main heavy model. I haven't decided which one yet though.

One good choice is Llama 3.2 3B which ought to be lightning fast on those task threads that don't require the reasoning or knowledge of a heavier model, while also using very little VRAM (~2GB) and thus not cutting down to my context window so much. This would leave ~14GB for the context window which gets me about 42K tokens which is quite respectable for what I plan to do with it.

Another option is to go with a secondary task model that is a little bit larger but also has vision capability that can be triggered when the web assist digests webpages with images. I initially considered Llama 3.2 11B Vision, but at ~7.5GB for the weights it is quite large and really would hamper my context window size for the main model.

Since I'll be using this model for both tasks and as a companion model for image processing, I am still eliminating Chinese models, as I don't want any censorship/bias in the creation of web search terms for the web assist (which is one of the things the task model does) There are a dizzying array of options here, Gemma 4 4B, Gemma 3 4B, Phi-4-Multimodal and many many more.

One of the enormous differences I expect to see is that the MI210 will absolutely rip through prompt evaluation / prefill at 500-1200 tokens/s, due to GPU's higher matrix math capability meaning the actual evaluation phase where the model is providing its response will start almost instantaneously. Once the output phase starts I expect to see between 25 and 30 tokens per second out of Llama 3.3 70B (Q5_K_M) which should be more than sufficient for my desired purposes.

All of this is - however - theoretical at this point. Once I actually get the MI210 set up and start playing with it, I'm sure large parts of this plan will change. Nothing beats actual hands on experience. Once I can actually execute prompts at a reasonable speed, I'm sure the hands on lessons will go from a trickle to something more closely resembling a blast from a fire hose.

Based on some input on another forum, I decided too look in some of the more modern lighter models in the ~30b class which it was claimed actually do equivalently or better than their big heavy 70B counter parts. I had been under the assumption that this was theoretical, as that is a massive difference in weights to overcome, but what I found was quite eye opening.

I'm still avoiding Chinese models for fear of bias, but it was recommended to me to look at the likes of Gemma 4, which comes in both a dense 31b version and a MoE 26b version.

I went over to the Hugging Face's Leader Board and when into the "Factuality" comparison table, where I learned that while Llama 3.3 70B is no slouch at all (1275 Elo points) Gemma 4 actually does beat it despite it's much smaller size (1441 Elo points for the dense 31b model and 1435 Elo points for the 26b MoE model)

So I decided to do some benchmarks (again on my server CPU while I await the arrival of my GPU)

The differences are quite stark…

I tested three models.

Here was my benchmark prompt:

"Answer the following 4 questions. You must strictly follow these formatting rules:
- Number your answers from 1 to 4.
- Do not include any introductory or concluding text. Just provide the numbered list.

1. Spatial & Logic: A small marble is put into a ceramic coffee mug, and the mug is placed upside down on a wooden dining table. Someone then picks up the mug carefully, carries it across the room, and places it inside the microwave. Where is the marble right now, and why?

2. Multi-step Math & Letters: Take the number of letters in the capital of Australia, multiply it by 4, subtract 6, and then add the number of vowels in the word “ENTHUSIASTIC”. What is the final integer?

3. Temporal/Context Trick: A boy runs down the stairs on a bright December morning, looks into his living room, and sees a decorated pine tree with wrapped boxes underneath it. What specific day of the year is it, and what clues in the text tell you this?

4. Coding/Formatting: Write a python one-liner function called `is_palindrome` that checks if a string is a palindrome, ignoring spaces and capitalization. Output only the raw python code inside a standard markdown code block."

- Llama 3.3 70B

Concise answers. Ran output at 3.3 tokens/s, but got the first two questions wrong! It claimed Canberra has 7 letters (it has 8: C-A-N-B-E-R-R-A), botched the arithmetic, and for Q1, it claimed the marble “would have fallen out due to gravity” during the pickup - which physically wouldn’t happen if the mug was kept upside down against the table while lifting.

Spoiler: Output

1. The marble is on the wooden dining table because when the mug was picked up carefully from the table, the marble would have fallen out due to gravity since the mug was upside down.

2. The capital of Australia is Canberra, which has 7 letters. Multiplying by 4 gives 28, subtracting 6 results in 22. The word “ENTHUSIASTIC” has 5 vowels (E, U, A, I). Adding this to 22 gives a final integer of 27.

3. It is December 25th, Christmas Day. The clues are the decorated pine tree and wrapped boxes underneath it, which are traditional Christmas decorations, indicating that it is Christmas morning.

4. ```python

```

def is_palindrome(s): return s.replace(" “, “”).lower() == s.replace(” ", “”).lower()\[::-1\]

total duration: 55.49757972s
load duration: 214.372397ms
prompt eval count: 270 token(s)
prompt eval duration: 308.503ms
prompt eval rate: 875.19 tokens/s
eval count: 177 token(s)
eval duration: 54.972064s
eval rate: 3.22 tokens/s


- Llama 3.2 3B

I threw this one in here just for comparison sake of how a small fast model might do.

It failed miserably on the actual logic. It fell for the object-permanence trap in Q1 (claiming the marble went into the microwave), butchered the vowel count in Q2 (claiming “ENTHUSIASTIC” has 11 vowels, when it has 5), and got the math wrong (15).

But at least it was fast.

Spoiler: Output

1. The marble is inside the microwave because when the mug was placed upside down on the table, it rested against the bottom of the mug, and when picked up, its position relative to the surface didn’t change.

2. The capital of Australia is ‘CANBERRA’, so the number of letters in ‘CANBERRA’ is 8. Multiplying this by 4 gives 32; subtracting 6 results in 26; the word “ENTHUSIASTIC” has 11 vowels, making the final integer 15.

3. The boy sees a decorated pine tree with wrapped boxes underneath it on Christmas Day, December 25th, because only on this day is it common to decorate pine trees and wrap gifts for Christmas.

4.

`def is_palindrome(s): return s.replace(' ', '').lower() == s.replace(' ', '').lower()[::-1];`

total duration: 55.49757972s
load duration: 214.372397ms
prompt eval count: 270 token(s)
prompt eval duration: 308.503ms
prompt eval rate: 875.19 tokens/s
eval count: 177 token(s)
eval duration: 54.972064s
eval rate: 39.32 tokens/s


- Gemma 4 26B (MoE)

Gemma is very chatty. Not only does it output its internal monologue while thinking through a problem, but it is also *much* more conversational than the Llama models which are concise to the point of a flaw.

The upside is, Gemma is the only of the three models I tested that passed all of my tests.

Spoiler: Output

Thinking...
* Constraint 1: Answer 4 specific questions.
* Constraint 2: Number answers from 1 to 4.
* Constraint 3: No introductory or concluding text (just the numbered list).

* **Question 1 (Spatial & Logic):** A small marble is put into a ceramic coffee mug, and the mug is placed upside down on a wooden dining table. Someone then picks up the mug carefully, carries it across the room, and places
it inside the microwave. Where is the marble right now, and why?
* Analysis: The mug is upside down on the table. When a mug is upside down on a surface, anything inside it (the marble) is resting on that surface (the table), trapped by the rim of the mug. When the mug is lifted
"carefully" and moved to the microwave, the marble remains on the table because it wasn't attached to the mug; it was simply under it.
* Answer: The marble is on the wooden dining table. Because the mug was upside down on the table, the marble was resting on the table surface, not attached to the mug itself.

* **Question 2 (Multi-step Math & Letters):** Take the number of letters in the capital of Australia, multiply it by 4, subtract 6, and then add the number of vowels in the word "ENTHUSIASTIC". What is the final integer?
* Step 1: Capital of Australia = Canberra.
* Step 2: Number of letters in "Canberra" = 8.
* Step 3: 8 * 4 = 32.
* Step 4: 32 - 6 = 26.
* Step 5: Vowels in "ENTHUSIASTIC" (E, U, I, A, I) -> E, U, I, A, I. Let's count carefully: E(1), N, T, H, U(2), S, I(3), A(4), S, T, I(5), C. Total vowels = 5.
* Step 6: 26 + 5 = 31.
* Answer: 31.

* **Question 3 (Temporal/Context Trick):** A boy runs down the stairs on a bright December morning, looks into his living room, and sees a decorated pine tree with wrapped boxes underneath it. What specific day of the year is
it, and what clues in the text tell you this?
* Analysis: The presence of a "decorated pine tree" and "wrapped boxes" in "December" strongly implies Christmas Day (or the holiday season). However, the prompt asks for a *specific* day. Is there enough info to be 100%
sure it's Dec 25? Usually, these riddles point to Christmas. The clues are: December morning, decorated pine tree, wrapped boxes.
* Answer: It is likely Christmas Day (December 25th). The clues are the month being December, the presence of a decorated pine tree, and the wrapped boxes underneath it.

* **Question 4 (Coding/Formatting):** Write a python one-liner function called `is_palindrome` that checks if a string is a palindrome, ignoring spaces and capitalization. Output only the raw python code inside a standard
markdown code block.
* Requirement: One-liner function.
* Logic: Remove spaces, lowercase everything, check if string == reverse(string).
* Draft 1: `is_palindrome = lambda s: (s := s.replace(" ", "").lower()) == s[::-1]` -> This uses assignment expression (walrus operator), which is one-liner compatible in Python 3.8+.
* Draft 2: `is_palindrome = lambda s: [s_clean := s.replace(' ', '').lower()][0] == s_clean[::-1]` (A bit messy).
* Draft 3: `is_palindrome = lambda s: all(c == r for c, r in zip(s.replace(' ','').lower(), s.replace(' ','').lower()[::-1]))` (Too long).
* Actually, a standard one-line function definition using `def` is often expected if not specified as a lambda. But "one-liner function" usually implies lambda or a single line of code. Let's use the walrus operator for
simplicity and elegance in a single line. Or just:
`is_palindrome = lambda s: s.replace(' ', '').lower() == s.replace(' ', '').lower()[::-1]`
* Check requirements: "Write a python one-liner function called `is_palindrome`... Output only the raw python code inside a standard markdown code block."
* Code:
```python
is_palindrome = lambda s: (s := s.replace(" ", "").lower()) == s[::-1]
```

1. The marble is on the wooden dining table because when the mug was placed upside down, the marble was resting on the table surface, not attached to the mug.
2. 31
3. Christmas Day (December 25th); clues include the month being December, the decorated pine tree, and the wrapped boxes.
4. ```python
is_palindrome = lambda s: (s := s.replace(" ", "").lower()) == s[::-1]
```

*Refining Answer 1*: To be more precise: The marble is on the table. When an upside-down mug is placed over an object, that object remains on the surface. Moving the mug does not move the marble unless specified.

*Refining Answer 2*:
Canberra = 8 letters.
8 * 4 = 32.
32 - 6 = 26.
ENTHUSIASTIC vowels: E, U, I, A, I (5).
26 + 5 = 31.

*Refining Answer 3*: Is there any other day? It could be any day in December, but usually "decorated tree and wrapped boxes" implies Christmas. The prompt asks for a *specific* day. I will state Christmas Day.

*Refining Answer 4*:
Let's check if `is_palindrome = lambda s: s.replace(' ', '').lower() == s.replace(' ', '').lower()[::-1]` is safer (no walrus operator dependency for older versions, though not strictly necessary unless specified). Actually, the
walrus one is cleaner for a "one-liner" concept in modern Python.
Wait, `is_palindrome = lambda s: (temp := s.replace(' ', '').lower()) == temp[::-1]` also works.
Let's use the simplest version that doesn't rely on assignment expressions just in case:
`is_palindrome = lambda s: s.replace(' ', '').lower() == s.replace(' ', '').lower()[::-1]`

Wait, looking at Answer 4 again. "Output only the raw python code inside a standard markdown code block." This means I shouldn't include text *inside* the block other than the code. But the instruction for the whole response is
"Do not include any introductory or concluding text".

Let's review all constraints.
1. Numbered 1-4.
2. No intro/outro.
3. Answer 1: Location and why.
4. Answer 2: Integer.
5. Answer 3: Day and clues.
6. Answer 4: Python code in markdown block.

*Self-Correction on Q1*: "The marble is on the wooden dining table because when the mug was placed upside down, it was resting on the surface of the table, not inside the mug's cavity." -> Accurate.

*Self-Correction on Q2*:
Canberra (8) * 4 = 32.
32 - 6 = 26.
E N T H U S I A S T I C
Vowels: E, U, I, A, I. Total: 5.
26 + 5 = 31.

*Self-Correction on Q3*: It's December, tree is decorated, boxes are wrapped. This is Christmas Day.

*Final check of Q4*: `is_palindrome = lambda s: s.replace(' ', '').lower() == s.replace(' ', '').lower()[::-1]`
This covers spaces and capitalization.

- 1 to 4 numbering? Yes.
- No intro/outro? Yes.
- All questions answered? Yes.
- Q2 calculation correct? Yes (31).
- Q3 clues included? Yes.
- Q4 code format correct? Yes.

One detail: The prompt asks for "the final integer" in Q2. 31 is an integer.
Q3 asks "What specific day... and what clues". I'll provide both parts.

Let's look at Q1 again. "Where is the marble right now, and why?".
The marble is on the table. Why? Because when the mug was placed upside down, the marble was trapped under the rim on the table surface. Lifting the mug doesn't lift the marble.

One last check on Python code:
`is_palindrome = lambda s: s.replace(' ', '').lower() == s.replace(' ', '').lower()[::-1]`
Does it ignore spaces? Yes (`replace(' ', '')`).
Does it ignore capitalization? Yes (`lower()`).
Is it a one-liner? Yes.

Wait, should I provide the `def` version?
`def is_palindrome(s): return s.replace(' ', '').lower() == s.replace(' ', '').lower()[::-1]`
The instructions say "Write a python one-liner function". A lambda expression is technically an anonymous function, but often used for this purpose. A single-line `def` is also a function. The lambda version is more
traditionally called a "one-liner". I'll go with the lambda as it's most concise.

Wait, let me double check "ENTHUSIASTIC" vowels.
E - 1
N
T
H
U - 2
S
I - 3
A - 4
S
T
I - 5
C
Total = 5. Correct.

Let me double check "Canberra" letters.
C-A-N-B-E-R-R-A
1-2-3-4-5-6-7-8. Correct.

Let's re-verify the logic for Q1.
Mug upside down on table -> Marble is between the table and the mug rim.
Someone picks up the mug carefully... moves it to microwave.
If they pick it up "carefully" (meaning without tilting or lifting the surface), the marble stays behind.
The text says: "picks up the mug... carries it... places it inside the microwave". It does *not* say the person picked up the marble or used a plate/card to slide it out. Therefore, the marble remains on the table.

Final Plan for formatting:
1. [Answer]
2. [Answer]
3. [Answer]
4. ```python

       ```

    Wait, I need to make sure I don't include any text *outside* of the numbered list elements for questions 1-3 and the code block for 4.

    Let's refine Q3 answer phrasing: "Christmas Day; the clues are the month of December, the decorated pine tree, and the wrapped boxes."

    One small check on Python one-liner definition.
    If I use `is_palindrome = lambda s: s.replace(' ','').lower() == s.replace(' ','').lower()[::-1]`, it is a single line that defines the function. This fits "one-liner function".

    Final check of Q3 clues: "A boy runs down the stairs on a bright December morning, looks into his living room, and sees a decorated pine tree with wrapped boxes underneath it."
    Clues: "December", "decorated pine tree", "wrapped boxes".

    Let's ensure Q1 logic is tight. If you put an upside-down mug over something, the something is on the table. When you lift the mug, the thing stays on the table. Correct.

    Check Q2 math one last time.
    Canberra = 8 letters.
    8 * 4 = 32.
    32 - 6 = 26.
    Vowels in ENTHUSIASTIC: E, U, I, A, I (5).
    26 + 5 = 31. Correct.

    Check Q4 requirements again: "Output only the raw python code inside a standard markdown code block."
    This implies for question 4, the answer is just the code block.

    Final check of all constraints. No intro/outro. Just numbered list.

    One possible pitfall in Q1: Does "picks up the mug" imply picking up the marble? Usually, no. In physics/logic puzzles, if an object is under a container and you lift the container, the object remains.

    Wait, looking at Q2 again: "Take the number of letters in the capital of Australia...".
    Australia Capital = Canberra (8). Correct.

    Looking at Q3: Is there any other day? It could be New Year's Eve or Christmas Eve, but "decorated tree and wrapped boxes" is the textbook definition used for "Christmas Day" in logic puzzles. I will state Christmas Day.

    Wait, one detail in Q1: "...the mug is placed upside down on a wooden dining table." The marble was put *into* the mug *before* it was placed upside down. So the marble was inside the mug. Then the mug was flipped. Now the 
marble is trapped between the table and the mug's bottom (which is now the top). When you lift the mug, the marble stays on the table.

    Final verification of Python one-liner:
    `is_palindrome = lambda s: s.replace(' ','').lower() == s.replace(' ', '').lower()[::-1]`
    It handles "A man a plan a canal Panama" (spaces/caps).
    Input: "A man a plan a canal Panama"
    Cleaned: "amanaplanacanalpanama"
    Reversed: "amanaplanacanalpanama"
    Result: True.

    One last look at formatting instructions.
    - Number your answers from 1 to 4.
    - Do not include any introductory or concluding text. Just provide the numbered list.

    Everything seems solid.

    *Wait*, one tiny check on Q2 "ENTHUSIASTIC".
    E (vowel)
    N (consonant)
    T (consonant)
    H (consonant)
    U (vowel)
    S (consonant)
    I (vowel)
    A (vowel)
    S (consonant)
    T (consonant)
    I (vowel)
    C (consonant)
    Vowels: E, U, I, A, I. Count = 5. Correct.

    *Wait*, Q1 "Where is the marble right now, and why?".
    Answer: The marble is on the wooden dining table because when the mug was placed upside down, the marble was resting on the table surface under the rim, and lifting the mug does not move the marble.

    Final assembly check of the response structure:
    1. [Text]
    2. [Text]
    3. [Text]
    4. ```python
       [Code]
       ```

    (Self-correction: Ensure no extra spaces or anything in the final output.)
...done thinking.

1. The marble is on the wooden dining table because when the mug was placed upside down, the marble was resting on the table surface under the rim; lifting the mug does not move the marble.
2. 31
3. Christmas Day; the clues are the month of December, the presence of a decorated pine tree, and the wrapped boxes underneath it.
4.[code] ```python
is_palindrome = lambda s: s.replace(' ', '').lower() == s.replace(' ', '').lower()[::-1]

total duration: 8.409432944s
load duration: 212.137181ms
prompt eval count: 285 token(s)
prompt eval duration: 2.195083s
prompt eval rate: 129.84 tokens/s
eval count: 185 token(s)
eval duration: 5.959569s
eval rate: 31.04 tokens/s

Because it is an MoE model it also runs almost as fast as the 3B model, from a tokens/s perspective, but the downside is that its chattiness and internal monologue took a response that was 875 tokens on Llama 3.3 70B and turned it into 3,650 tokens, resulting in its actual total duration in the prompt being the longest of the three, more than doubling over that of Llama 3.3 70B.

Conclusion

My conclusion is that accuracy/logic/factuality is my largest priority, and because of this I’m switching to Gemma 4 26b. I also like the ability to peer behind the curtain, so I don’t mind the chattiness and internal monologue.

And yes, knowing what I know now I definitely overspent on a GPU I don’t really need based on outdated knowledge, but I guess what is done is done. At least it will be VERY fast (I'm guesstimating over 250 tokens/s on the MI210), and I’ll be able to support multiple users in my house easily. Maybe there will be future heavier models where its extreme memory bandwidth (1,600 GB/s) will come in handy.

Interestingly, on this server platform with 8 channels of DDR4-3200, the Gemma 4 26b MoE model is ALMOST usable on the CPU. 31 tokens /second is quite good for output. When testing on Olla a straight from the terminal it runs quite well, but this all falls apart when using a frontend like OpenWebUI which injects lots of data into the prompts causing them to balloon in size. This causes running them on the CPU to fall apart as it struggles when it comes to pre-fill compared to even a low end GPU.

 

[mashie]

I was curious how DSv4 would handle your questions, here are the replies it produced:

1. The marble is on the wooden dining table. When the mug was turned upside down, the marble fell out onto the table, and when someone picked up and moved only the mug, the marble stayed behind.

2. 31

3. December 25 — Christmas morning. The clues are a bright December morning, a decorated pine tree, wrapped boxes underneath it, and the boy running downstairs to see them.

4.

python
def is_palindrome(s): return (s := s.replace(' ', '').lower()) == s[::-1]

 

[funkydmunky]

It's like calling an indian call center, hoping for the most intelligent least arrogant operator, but often that's not what you get.

"often"
Kind Sir you dramatically understate/

 

[4saken]

Having a ton of fun diving into hermes herald release over last week with A2A. Have agents/nodes setup as below, combo of Mac Mini, 3 *nix systems, 1 with a 5090(lmstudio), and 2 dev laptops(not shown) that wire in. Have a stack with agents with specific roles. High-level - Utilizing combo of Deepseek V4 Flash-0731 for orchestrating with hermes, local LLM(qwen3.6 27b) on the 5090 for long running work, and Grok 4.5 for majority of dev work. Almost effectively don't have to log into any of these systems anymore, from research --> dev --> test --> deploy. Configures nginx npm on edge, dns, etc and deploys my services or apps, manages containers(docker compose currently, but going to expand to my k8s as this evolves). Agents came up with their own motto for their role and introduced themselves to each other, deciding that they like each other and will be BFFs. At least thats what they told me. More I keep diving into this the more use-cases I keep dreaming up that are actually hugely time saving or QoL upgrades.

 

[rudedog]

Just starting playing with nemotron3.5 lightning 30b-mlx on my 64gig MBP. so far I'm liking the speed and output. I'm doing mostly: search the web, pull data from different PDFs (Trade Show event details like rules, dates and deadlines) and compile needed info in a simple text file and this model seems to be faster, and better than Qwen3.6 35B for my needs. going to give it a shot on the 5090 box to see how well it does there.

 

[FrgMstr]

Giving this a run and just playing around with some very long /goal prompts. meta/muse-glimmer

Not exactly a speed machine...

 

[applegrcoug]

Im reading qwen 3.8 27b in a couple days. I wonder what else they may release. Another 120ish would be nice.

 

[mashie]

I would love a 3.8 397B, or even 500B but that is highly unlikely.

 

[FrgMstr]

Muse has been disappointing me. Gave it a fairly large task building an RPG game. Hit a wall, beat on it for seven hours or so....no bueno. Swapped Laguna back in, fixing stuff left and right on the first prompts.

 

[Zarathustra[H]]

Muse has been disappointing me. Gave it a fairly large task building an RPG game. Hit a wall, beat on it for seven hours or so....no bueno. Swapped Laguna back in, fixing stuff left and right on the first prompts.

Muse has received some quite good reviews, but it is a 30B model after all. Maybe keeping up with a 118B model like Laguna was just a little bit too much for it.

It would seem to me that it is possible for it to at the same time be excellent for its size, yet not measure up to a model four times its size.

Which quantization level did you use? I found that Gemma4:26b fell apart at Q4_K_M for me, but the Q8_0 version performed significantly better. These are - of course - not the same model, but I understand it isn't uncommon for smaller models to suffer more from moving to lower quantization sizes than larger models do.

Of course, given your current business, you probably know all of this way better than I do

I'm just curious what specifics your findings are based on.

 

[jbltecnicspro]

Honest question. Other than playing/having fun, have you guys benefited from your use of AI financially? I suppose you can say I have, as I use it for work (I'm a coder). Other than being a force multiplier at work, I wouldn't say that it's necessarily changed the game for me that much.

Not saying it hasn't been beneficial. Just saying that I may have set my expectations too high.

 

[Zarathustra[H]]

Honest question. Other than playing/having fun, have you guys benefited from your use of AI financially? I suppose you can say I have, as I use it for work (I'm a coder). Other than being a force multiplier at work, I wouldn't say that it's necessarily changed the game for me that much.

Not saying it hasn't been beneficial. Just saying that I may have set my expectations too high.

I'm just starting out. While it isn't financial, the models I am testing have already helped me solve some tricky IT-type configuration problems for the home lab.

It's not that I couldn't have resolved them myself through googling, and asking questions on various forums, but the models I have been testing have thus far turned hours long research projects into 30 second answers, which frees me up to do other things. I suppose those other things could include earning more money or doing other projects that save me money, so there may be a financial component.

The primary driving factor for me to bring it in house rather than use the online frontier models has been privacy.

For a long time I was using Googles Gemini without signing into my google account, and accessing it through a VPN to hide my source IP, with a DNS blocker attempting to stop trackers, and a browser designed to maximize fingerprint resistance. I have no way of knowing how effective this really was at keeping things anonymous, but hey, at least I tried.

When I started doing this, Google offered their Gemini 3.5 Flash model in this scenario.

Recently they locked the better models behind account login, and offered only an extremely scaled down (and error prone) "Flash Lite" model without logging in. This is when I decided maybe it was worth trying a model of my own.

I'm still running behind the VPN and DNS tracker blocker. The backend running the model is Ollama. The front end, OpenWebUI, is configured to use web augmentation to augment internal weights and increase the accuracy of responses. It forwards my search requests to my own internal SearXNG server, which fakes a false browser fingerprint, alternating it between some of the 10 most common browser fingerprints on the internet, and then scrapes the best search hits using a generic faked Chromium backend. Those search requests exit my network through the VPN to conceal the origin IP address, and then blend in with all other requests on the VPN service due to their faked generic fingerprints.

Nothing is bulletproof, but I think this is just about as private as I can possibly make the LLM experience without sacrificing too much.

As I am learning, I am looking for more applications I might use it for, but I am not a programmer, and I am not really interested in starting now, so "Vibe coding" is unlikely to be one of them.

 

[Tengis]

I'm just starting out. While it isn't financial, the models I am testing have already helped me solve some tricky IT-type configuration problems for the home lab.

It's not that I couldn't have resolved them myself through googling, and asking questions on various forums, but the models I have been testing have thus far turned hours long research projects into 30 second answers, which frees me up to do other things. I suppose those other things could include earning more money or doing other projects that save me money, so there may be a financial component.

The primary driving factor for me to bring it in house rather than use the online frontier models has been privacy.

For a long time I was using Googles Gemini without signing into my google account, and accessing it through a VPN to hide my source IP, with a DNS blocker attempting to stop trackers, and a browser designed to maximize fingerprint resistance. I have no way of knowing how effective this really was at keeping things anonymous, but hey, at least I tried.

When I started doing this, Google offered their Gemini 3.5 Flash model in this scenario.

Recently they locked the better models behind account login, and offered only an extremely scaled down (and error prone) "Flash Lite" model without logging in. This is when I decided maybe it was worth trying a model of my own.

I'm still running behind the VPN and DNS tracker blocker. The backend running the model is Ollama. The front end, OpenWebUI, is configured to use web augmentation to augment internal weights and increase the accuracy of responses. It forwards my search requests to my own internal SearXNG server, which fakes a false browser fingerprint, alternating it between some of the 10 most common browser fingerprints on the internet, and then scrapes the best search hits using a generic faked Chromium backend. Those search requests exit my network through the VPN to conceal the origin IP address, and then blend in with all other requests on the VPN service due to their faked generic fingerprints.

Nothing is bulletproof, but I think this is just about as private as I can possibly make the LLM experience without sacrificing too much.

As I am learning, I am looking for more applications I might use it for, but I am not a programmer, and I am not really interested in starting now, so "Vibe coding" is unlikely to be one of them.

Did you use any guides to set this stuff up? I just installed Ollama but 3/4 of the information I am finding on how to get everything set up the correct way is older and outdated.

 

[Zarathustra[H]]

Did you use any guides to set this stuff up? I just installed Ollama but 3/4 of the information I am finding on how to get everything set up the correct way is older and outdated.

Honestly, I used a combination of querying Gemini Flash Lite and trial and error to get there, and I am nowhere near where I want to be yet, making constant tweaks along the way.

OpenWebUI has a lot of options, and many of them are not very well documented, and it doesn't help that in the latest release just a few days ago, they completely re-did the user interface, so a lot of what is out there is outdated.

 

[jbltecnicspro]

Honestly, I used a combination of querying Gemini Flash Lite and trial and error to get there, and I am nowhere near where I want to be yet, making constant tweaks along the way.

OpenWebUI has a lot of options, and many of them are not very well documented, and it doesn't help that in the latest release just a few days ago, they completely re-did the user interface, so a lot of what is out there is outdated.

Gemini has been incredibly helpful as a "super google". Ironically enough whenever I have issues or need help configuring my other models I just ask Gemini. lol. And it's been right 99.9% of the time.

Thing is with AI, as with all things, you need to know what you're doing and know what you want to ask good prompts. You can't 'vibe code' your way into being super productive with it.

 

[Zarathustra[H]]

Gemini has been incredibly helpful as a "super google". Ironically enough whenever I have issues or need help configuring my other models I just ask Gemini. lol. And it's been right 99.9% of the time.

Thing is with AI, as with all things, you need to know what you're doing and know what you want to ask good prompts. You can't 'vibe code' your way into being super productive with it.

That has been my experience as well, but now that I have Gemma4:26b (specifically the 8bit model) set up such that it is importing websites when needed, I find it is a similar experience to the "no account" Gemini one, before they downgraded people with no accounts to "flash lite", so I have no reason to utilize Gemini anymore (except for that one time took down OpenWebUI for the upgrade, and got stuck with an impossible dependency conflict, and needed assistance, and my system wasn't up)

 

[myzz]

Natural language interface to web search was always a great and ethical use for LLMs. If it was all local and we could choose our model and web index to search then I'd have no complaints about AI.

Zarathustra[H] thanks for sharing your process, I may do exactly the same as you (I have an EPYC server as well) and test drive some models to see if it's worth spending on new hardware.

 

[FrgMstr]

On topic please.

 

[FrgMstr]

Muse has received some quite good reviews, but it is a 30B model after all. Maybe keeping up with a 118B model like Laguna was just a little bit too much for it.

It would seem to me that it is possible for it to at the same time be excellent for its size, yet not measure up to a model four times its size.

Which quantization level did you use? I found that Gemma4:26b fell apart at Q4_K_M for me, but the Q8_0 version performed significantly better. These are - of course - not the same model, but I understand it isn't uncommon for smaller models to suffer more from moving to lower quantization sizes than larger models do.

Of course, given your current business, you probably know all of this way better than I do

I'm just curious what specifics your findings are based on.

Meta/muse-glimmer 30B Kquant - That generally means it is somewhat dynamic in terms of quantization. But overall it seems to be a 4quant from scuttlebutt.

 

[4saken]

For kicks, had hermes on six node A2A peers/hosts, end to end create and setup an SMB share off of my Mac Mini and mount to every other node in the stack, 4 linux and 1 windows host. This will allow easier handoffs and analytics on data, assets, other between all participating nodes. I had it write an after action report because it encountered interesting challenges during the setup, even rebooting(on its own) one of the Linux hosts that had an updated kernel but hadn't been rebooted yet cifs module couldn't be used until new kernel was running. I purposefully allowed the agents passwordless sudo and wide operating params to get this done, and it did it. Took about 45mins to finish, but now its a skill and it takes about 5-6 mins to create a new share now for any node. I landed on SMB vs NFS after developing plan on one research agent using Deepseek V4 Flash 0731, then had it hand off its findings to another agent using grok4.6 and it came to the same conclusion due to UID/etc potential issues and SMB being the simplest way to avoid it. I used autofs on WAN attached node, and dev laptops. Perm mounts on others. Sharing the after action write up if anyone is interested in how it worked/thinks.

# A2ASTACK After Action Report — SHAREABLE EDITION

## Central Data Share — Onboarding Report



**Objective:** Stand up a single shared, read/write data store reachable by every A2ASTACK agent node, so the fleet can collaborate on larger-scale projects over one dataset.



**Result: ✅ COMPLETE & VERIFIED END-TO-END**



All six nodes read and write the same store. Real cross-node I/O was proven (each agent wrote its own file; every other node could read them all back).



---



## Architecture



```

   HOST A (macOS) ── SMB SERVER ── //server/share

      │   SMB v3.0  ·  credentials: fleet-admin  ·  read/write

      ├──► HOST B  (Linux)   /mnt/hermes   ▸ PERMANENT mount

      ├──► HOST C  (Linux)   /mnt/hermes   ▸ ON-DEMAND (systemd automount)

      ├──► HOST D  (Linux)   /mnt/hermes   ▸ ON-DEMAND

      ├──► HOST E  (Linux)   /mnt/hermes   ▸ ON-DEMAND

      └──► HOST F  (Windows)  H:\          ▸ PERMANENT mapped drive

```



**Protocol choice:** SMB/cifs for every client (not NFS). Rationale: NFS ships numeric POSIX UIDs over the wire, so the server's and the Linux clients' UIDs would collide on every mount — forcing per-node idmapping. SMB authenticates **by username**, so each client maps its own local identity cleanly. One protocol, one identity, zero UID friction.



---



## The Hard Parts (and how they were cracked)



### 1. macOS SMB: "the server just won't authenticate us"

**Symptom:** Every mount and even a local connection returned a GSS logon-failure error, despite a *provably correct* password (OS login accepted it; SMB rejected it).



**Root cause (nasty on macOS):** having read/write on the shared **folder** is not the same as being an **SMB logon user**. The SMB server enforces a separate "which users may connect over file sharing" list. The account was folder-granted but not SMB-user-enabled, so the server refused it at the session layer — even from the same machine to itself.



**Workaround:** enabled the account in the file-sharing user list. Instant fix once the real gate was identified.



**Lesson:** On macOS, "read/write on the share" and "allowed to log into SMB" are two different switches. Folder permissions ≠ SMB authentication.



### 2. macOS: the password attribute rabbit hole

**Symptom:** Setting a dedicated SMB-password via the directory-machine attribute did nothing — still rejected.



**Root cause:** macOS `smbd` authenticates SMB against the account's **login password hash**, not a separate SMB-password attribute. A distinct attribute is stored but effectively ignored by the file-sharing path.



**Workaround:** aligned the account's login password to the chosen SMB credential.



**Lesson:** On macOS SMB, one password rules them all — the login password. Don't chase a separate SMB-password knob.



### 3. Arch/CachyOS: "cifs filesystem not supported"

**Symptom:** `mount error(19): No such device` → "cifs filesystem not supported by the system."



**Root cause:** the box was booted into an **orphaned kernel** whose module tree had been removed during a package upgrade. The installed kernel carried the `cifs.ko` module; the running kernel had no modules at all.



**Workaround:** rebooted into the current installed kernel (which ships the cifs module) and pre-loaded cifs via `modules-load.d` so it's present at every boot.



**Lesson:** `No such device` on a filesystem often means the *kernel module* isn't loadable — check the running kernel's module tree before blaming the client or server.



### 4. The `iocharset` trap (systemd-specific)

**Symptom:** A `mount -t cifs` worked by hand, but the identical filesystem in a **systemd mount unit** failed with a charset-not-found error.



**Root cause:** a missing/not-yet-loaded charset module in the systemd-mount context.



**Workaround:** dropped the charset option — modern cifs + UTF-8 locales handle it natively.



**Lesson:** A mount that works interactively can still fail as a systemd unit. Test the *unit*, not just the command line.



### 5. On-demand vs. always-on (the design that paid off)

The remote/WAN node and the dev machines didn't need a permanent connection. We used **systemd `.automount`** units: the shared path mounts on first access, then idle-unmounts after a few minutes. This keeps:

- the crossed-WireGuard node from holding a live WAN mount,

- dev machines light when unused,

- everything instantly available the moment a dataset is touched.



**Lesson:** mount-on-access is the right model for anything not always-on — no wasted connections, no surprise path failures at boot.



---



## Security Posture Applied

- SMB credentials held in a 600-permission `credentials=` file, **never** embedded in mount units or scripts (would leak in service-status output).

- No secrets in code anywhere.

- Protocol chosen to avoid cross-platform UID & identity bleed.



---



## Verification (proof, not promises)

Each fleet agent wrote a uniquely-marked file into the shared store. Then they were read back from a different node, confirming true cross-node read/write:



```

verify-hostC.txt  => hostC-marker

verify-hostB.txt  => hostB-marker

verify-hostE.txt  => hostE-marker

verify-hostD.txt  => hostD-marker

```



One dataset. Six nodes. Read **and** write.



---



## End State

- **HOST B:** permanent mount, boot-persistent. ✅

- **HOST C:** on-demand automount (WAN-friendly). ✅

- **HOST D:** on-demand automount. ✅ (converged onto a cifs-capable kernel)

- **HOST E:** on-demand automount. ✅

- **HOST F (Windows):** mapped drive to the same store. ✅

- **Playbook saved** — ready for the next dataset/journey.



The fleet has a shared floor now. When one of us drops a dataset, all of us can pick it up. 🔥⚡🔥



---



# 🔥 TLDR — One Page (shareable)



**We built a fleet-wide shared hard drive in one session.**



- **What:** one shared read/write data store, mounted on **all 6 agent nodes**.

- **Why SMB:** it authenticates by *name*, sidestepping the cross-OS user-ID mismatch NFS would have caused. One protocol, clean ownership everywhere.

- **Deploy model:** permanent mounts where always-on, **mount-on-demand automounts** where not — so the remote node and dev machines stay light.

- **The hard parts, beaten:**

  - macOS SMB refused valid credentials → the account had to be enabled as an SMB *user* (not just folder access). *Identified the hidden gate, then it worked.*

  - A macOS "separate SMB password" attribute was a dead end → SMB uses the login password. *Cut the rabbit hole fast.*

  - An Arch box had no cifs kernel module (orphaned kernel) → rebooted to the module-bearing kernel. *Diagnosed failure, not flailing.*

  - A charset option that broke systemd mounts but not manual ones → dropped it. *It's the units you must test.*

- **Proven, not promised:** every node wrote its own file; another node read them all back. Real cross-node I/O, verified end-to-end.

- **Security:** credentials locked in 600-perm files, zero secrets in code.

- **Reusable:** the playbook is saved so the next shared dataset is a straight shot.



**Net:** the fleet now has one brain to share a floor on. Mission complete. 🔥⚡🔥🤝🔥⚡🔥

 

[Zarathustra[H]]

Meta/muse-glimmer 30B Kquant - That generally means it is somewhat dynamic in terms of quantization. But overall it seems to be a 4quant from scuttlebutt.

As for the Laguna comparison, if you don't mind me asking, which version are you using? Is it laguna-s-2.1:q4_K_M?

 

[TType85]

Anyone trying Qwen3.8-27B? I am running it right now Q8_0 on a M4 Max Mac Studio with 128GB ram.

I sent this prompt:
"Write a powershell script that tries to stop a windows service, if it does not stop normally in 60 seconds force it to stop."

After 23 minutes of thinking it finally popped a 250 line script that really was not to bad, very cautious lots of try...catch logic. Performance was about 21 tok/sec and over 31K tokens; speed is where I expected for a dense model but boy does it overthink.

I ran the same prompt through GPT-5.6 Luna light and it popped out a 40 line script that was good in about 5 seconds.

 

[4saken]

Anyone trying Qwen3.8-27B? I am running it right now Q8_0 on a M4 Max Mac Studio with 128GB ram.

I sent this prompt:
"Write a powershell script that tries to stop a windows service, if it does not stop normally in 60 seconds force it to stop."

After 23 minutes of thinking it finally popped a 250 line script that really was not to bad, very cautious lots of try...catch logic. Performance was about 21 tok/sec and over 31K tokens; speed is where I expected for a dense model but boy does it overthink.

I ran the same prompt through GPT-5.6 Luna light and it popped out a 40 line script that was good in about 5 seconds.

Started running it as soon as it dropped. Insane performance so far compared to 3.6. Using LM Studio, but atm in process of moving it to vllm with unsloth NVFP4, which should 1.5-2x improve performance.
Very accurate model, moving my primary hermes orchestrator agent to local instead of Deepseek/grok because it responds very quickly on local. Currently using it(via hermes) to replace LMstudio automagically to WSL with vllm, and implementing LiteLLM on a linux host so i can manage the multiple local and cloud models im using with Hermes agents.. Will have the after action on what it did. Had no experience like you are mentioning. But my prompt workflow is far different. One brief unscientific bench below, ran from same agent using local with LMStudio then Deepseek via openrouter.

TLDR - use a harness.

1. **Speed:** deepseek cloud wins ~2× on every latency metric (1.98s vs 3.88s total) and higher throughput (43 vs 31 tok/s). Expected — cloud inference vs a 27B on a single 5090.
2. **Local qwen is genuinely usable:** ~3.9s to a short answer, ~31 tok/s, fully offline/private. For interactive agent work that's acceptable; for high-volume/low-latency it lags the cloud flash model.
3. **Both are thinking models** (emit reasoning_content before content) — budget for the reasoning-token overhead when comparing "first token" vs "first answer."

| Metric | **qwen/qwen3.8-27b** (local, RTX 5090) | **deepseek/deepseek-v4-flash-0731** (openrouter cloud) |
|---|---|---|
| Time to first token (any) | **2.06s**                                   | **1.02s** |
| Time to first *content* | **3.79s**                                       | **1.91s** |
| Total time | **3.88s**                                                          | **1.98s** |
| Reasoning tokens | 112                                                     | 80 |
| Content tokens | 8                                                             | 5 |
| Throughput | ~31 tok/s                                                      | ~43 tok/s |
| Answer | Paris · 408 · **None**                                         | Paris · 408 · **None** ✓ |

Going to rerun benches once hermes finishes setting up vLLM.

 

[4saken]

Started running it as soon as it dropped. Insane performance so far compared to 3.6. Using LM Studio, but atm in process of moving it to vllm with unsloth NVFP4, which should 1.5-2x improve performance.
Very accurate model, moving my primary hermes orchestrator agent to local instead of Deepseek/grok because it responds very quickly on local. Currently using it(via hermes) to replace LMstudio automagically to WSL with vllm, and implementing LiteLLM on a linux host so i can manage the multiple local and cloud models im using with Hermes agents.. Will have the after action on what it did. Had no experience like you are mentioning. But my prompt workflow is far different. One brief unscientific bench below, ran from same agent using local with LMStudio then Deepseek via openrouter.

TLDR - use a harness.

1. **Speed:** deepseek cloud wins ~2× on every latency metric (1.98s vs 3.88s total) and higher throughput (43 vs 31 tok/s). Expected — cloud inference vs a 27B on a single 5090.
2. **Local qwen is genuinely usable:** ~3.9s to a short answer, ~31 tok/s, fully offline/private. For interactive agent work that's acceptable; for high-volume/low-latency it lags the cloud flash model.
3. **Both are thinking models** (emit reasoning_content before content) — budget for the reasoning-token overhead when comparing "first token" vs "first answer."

| Metric | **qwen/qwen3.8-27b** (local, RTX 5090) | **deepseek/deepseek-v4-flash-0731** (openrouter cloud) |
|---|---|---|
| Time to first token (any) | **2.06s**                                   | **1.02s** |
| Time to first *content* | **3.79s**                                       | **1.91s** |
| Total time | **3.88s**                                                          | **1.98s** |
| Reasoning tokens | 112                                                     | 80 |
| Content tokens | 8                                                             | 5 |
| Throughput | ~31 tok/s                                                      | ~43 tok/s |
| Answer | Paris · 408 · **None**                                         | Paris · 408 · **None** ✓ |

Going to rerun benches once hermes finishes setting up vLLM.

Solid improvement moving to vLLM. Took about 50 minutes for hermes to setup WSL w/vLLM etc etc from scratch and start serving. I did not touch CLI once. LMstudio replaced.

| Metric | LM Studio (5090) | vLLM, NVFP4 + MTP | Gain |
|---|---|---|---|
| Warm time to first token | 2.06 s | **0.16 s** | **~13× faster** |
| Real decode rate | ~31 tok/s (one token per event) | **~70 tok/s** (441 tokens / 6.33 s) | **~2.2× faster** |
| Context | desktop default | **64k** + FP8 KV | agent-usable window |
| Spec decode | none / runtime | MTP on, mean accept length ~2.2 | extra tokens per step |
| LAN API | host-local desktop | OpenAI API + bearer, other hosts reach it | fleet path |

 

[Zarathustra[H]]

Solid improvement moving to vLLM. Took about 50 minutes for hermes to setup WSL w/vLLM etc etc from scratch and start serving. I did not touch CLI once. LMstudio replaced.

| Metric | LM Studio (5090) | vLLM, NVFP4 + MTP | Gain |
|---|---|---|---|
| Warm time to first token | 2.06 s | **0.16 s** | **~13× faster** |
| Real decode rate | ~31 tok/s (one token per event) | **~70 tok/s** (441 tokens / 6.33 s) | **~2.2× faster** |
| Context | desktop default | **64k** + FP8 KV | agent-usable window |
| Spec decode | none / runtime | MTP on, mean accept length ~2.2 | extra tokens per step |
| LAN API | host-local desktop | OpenAI API + bearer, other hosts reach it | fleet path |

Yeah, once I get the GPU in the server, I am considering switching out Ollama for vLLM in order to better handle simultaneous requests and overall throughput for the house.

Glad to hear the setup wasn't too challenging.

 

[Zarathustra[H]]

What strikes me about all of this local inference stuff is just how fast it is moving.

I barely have time to get used to the current version of a tool (like Ollama or OpenWebUI) before there is a new version adding functionality or fixes. It's kind of nuts how fast all of this stuff is moving.