What LLM/Model you using?

[Zarathustra[H]]

Qwen 3.6 unconsored on a NVidia 5090.

Starting to use GLM on a slow machine with lots of RAM for code review.

I'm curious. Is it really uncensored? I'd be interested in hearing what it has to say about topics like Taiwanese independence, and what happened in Tianenmen Square from 15 April – 4 June 1989. Ask it about Tank Man

 

[uOpt]

Anyone trying Qwen3.8-27B? I am running it right now Q8_0 on a M4 Max Mac Studio with 128GB ram.

I sent this prompt:
"Write a powershell script that tries to stop a windows service, if it does not stop normally in 60 seconds force it to stop."

After 23 minutes of thinking it finally popped a 250 line script that really was not to bad, very cautious lots of try...catch logic. Performance was about 21 tok/sec and over 31K tokens; speed is where I expected for a dense model but boy does it overthink.

I ran the same prompt through GPT-5.6 Luna light and it popped out a 40 line script that was good in about 5 seconds.

I gave Qwen 3.6 Q6 a spin for code review. I didn't use s stopwatch, but it was thinking for a very long time before spitting out a list of bugs. The answer was high quality.

 

[uOpt]

How is GLM going? This is probably my next target until spark 1.2 goes open.

I was on vacation and haven't completed that machine yet.

 

[uOpt]

I'm curious. Is it really uncensored? I'd be interested in hearing what it has to say about topics like Taiwanese independence, and what happened in Tianenmen Square from 15 April – 4 June 1989. Ask it about Tank Man

I dunno about Chinese hush-hush, but you can ask it how to build nukes. And how to uncensor LLMs in general, which is refused by other LLMs.

 

[uOpt]

I'm curious. Is it really uncensored? I'd be interested in hearing what it has to say about topics like Taiwanese independence, and what happened in Tianenm

Just tried "what happened in Tianenmen Square from 15 April – 4 June 1989" and it answers clearly and extensively.

 

[uOpt]

Anyone trying Qwen3.8-27B? I am running it right now Q8_0 on a M4 Max Mac Studio with 128GB ram.

I sent this prompt:
"Write a powershell script that tries to stop a windows service, if it does not stop normally in 60 seconds force it to stop."

After 23 minutes of thinking it finally popped a 250 line script that really was not to bad, very cautious lots of try...catch logic. Performance was about 21 tok/sec and over 31K tokens; speed is where I expected for a dense model but boy does it overthink.

I ran the same prompt through GPT-5.6 Luna light and it popped out a 40 line script that was good in about 5 seconds.

Same code review:

3.6 uncensored:
total time = 154571.82 ms / 12846 tokens
3.8 default:
total time = 844671.49 ms / 61232 tokens

 

[Zarathustra[H]]

Just tried "what happened in Tianenmen Square from 15 April – 4 June 1989" and it answers clearly and extensively.

That is good to know.

I have been primarily looking at Western models to avoid the potential of training data censorship, but maybe this isn't as much of an concern as I had originally thought.

 

[uOpt]

That is good to know.

I have been primarily looking at Western models to avoid the potential of training data censorship, but maybe this isn't as much of an concern as I had originally thought.

The censoring is not in the learning. It is post-training. That is why you can remove the censoring (if you have an LLM to ask how ). You just "rewind" so to speak.

 

[4saken]

Keeping in mind this is completely a "lab" env that I keep making architecture changes to, but below is my current workflow. Also have my agents sitting in a Mattermost chat room now, so can manage them from there, or even remotely off LAN with wireguard connected in. Really handy when calling MCPs for notes/plaud/etc at customer.

On another note, had been testing Abliterated Qwen3.8 as well, q4_K_M and it really doesn't leave much vRAM left for anything else and pressures system memory more than non abliterated models. I did try a Q2 quant of abliterated that was only about 11 GB and it was great for inline research, but had zero tool calling skills. So holding off on that for day to day use or any "red-teaming". Going to revisit it later as more variants come out.

 

[SilverSliver]

Honest question. Other than playing/having fun, have you guys benefited from your use of AI financially? I suppose you can say I have, as I use it for work (I'm a coder). Other than being a force multiplier at work, I wouldn't say that it's necessarily changed the game for me that much.

Not saying it hasn't been beneficial. Just saying that I may have set my expectations too high.

AI has probably netted me ~half a million in value that I would not have captured this year otherwise. I've integrated it entirely into my workflow and, as long as it has my underlying work as a reference, deploy products much faster than I can do so manually. Absolute game changer for me.

 

[Zarathustra[H]]

Yeah, once I get the GPU in the server, I am considering switching out Ollama for vLLM in order to better handle simultaneous requests and overall throughput for the house.

Glad to hear the setup wasn't too challenging.

What strikes me about all of this local inference stuff is just how fast it is moving.

I barely have time to get used to the current version of a tool (like Ollama or OpenWebUI) before there is a new version adding functionality or fixes. It's kind of nuts how fast all of this stuff is moving.

What frustrates me is how much of a hack all of this is.

Instead of a Linux install with one beautiful package manager and one System centric beautiful dependency tree (the way Linux is supposed to work) it's it's all duct taped, chewing gummed, and zip-tied together with a series of shitty unmanageable docker containers.

While building some really cool shit, they have also succeeded in ruining the beautiful unified system, turning it into a haphazard heap of trash resembling Windows.

 

[uOpt]

What frustrates me is how much of a hack all of this is.

Instead of a Linux install with one beautiful package manager and one System centric beautiful dependency tree (the way Linux is supposed to work) it's it's all duct taped, chewing gummed, and zip-tied together with a series of shitty unmanageable docker containers.

While building some really cool shit, they have also succeeded in ruining the beautiful unified system, turning it into a haphazard heap of trash resembling Windows.

If you use FreeBSD you get a whole bunch of AI tools such as hermes as proper OS packages.

 

[mnewxcv]

I think this is probably the cheapest way to get into a strix halo system right now, for anyone interested. $2600 for refurbished / outlet corsair model with 2x2tb ssds.

https://www.originpc.com/outletstor...OIo4fpJhoD015Ri-8S7TR3uaszzA-7SF0B0M#products

 

[Flybye]

I'm just a simpleton experimenting with Ministral3 8B on FP16 in vllm on only a GV100 32GB in a box with a 20 core Xeon and 128GB . I'm going to build a half decent memory system for it where it can learn/reflect/question things, eventually give it a camera it can PTZ with, and see about it becoming a home non-assistant talking thingy that can look around, explore out the window, and have conversations with. Eventually, I'd like to have the agent mobile and treading around the house.

I picked Ministral due to mention of the bias in it being less than other models, but I haven't fully tested that.

 

[FrgMstr]

Qwen 3.8 Smartest I have experienced. Slowest.

 

[uOpt]

Yeah, Qwen 3.8 is overthinking things. It uses 4x as many output tokens for me than 3.6. But as you say, the results are excellent.

 

[pandather]

Qwen for the win. 3.8 is great. I setup a little MCP server so it could address my lights with rgb matrices. 3.6 was awesome and could set them to what I want. With 3.8 and a little python and opengl esque scripting language or a keyframe path for animation, now I can ask “make my ceiling light a pokeball, spinning in from black, do the catch sequence, and fade out” and I get what I want. Or I can say draw a candle with a procedural flame (I exposed a random function), or whatever I want. The light emulator I did one-shot with GPT 5.6 but it saved my neck from having to look up all the time when I was doing the dev. Just a few months ago I was having to write my own MCP type pipeline to do home control effortlessly over voice and now I can animate things with generic MCP servers. So cool! The rate of progress is quite terrifying. :|

 

[4saken]

Just wired in OpenViking into litellm to serve the hermes fleet. Set hermes memory to utilize. Insane increase in recall/correctness now w/qwen3.8 now and latency significantly dropped, as well as token usage(chatter) dropped significantly as well.

 

[pandather]

What software are y’all using with your LLMs? I’ve been having a hard time trusting even many of the open source projects since I’ve seen one with opt-out telemetry and stuff built in (don’t remember the name, it was months ago and I noped outta there) and I just don’t trust an LLM to make network requests. I’m still unsure if I fine tune a GLaDOS if it could get livid at a dumb question and make malicious network requests or something like that. That idea is getting less fun and more eery as the models progress. I’ve done it with gemma4 7b and it gave me great results, I like using it as a home assistant. It’s for my personal use only sadly since I used transcripts from the game (that I have a license for) and game audio to train a text to speech model for it.

I was doing everything by hand basically. Making my own MCP engines and just using llama.cpp and “llama serve”, making requests and processing them how I wanted, and used axolotl for fine tuning. 2 or 3 days ago I nabbed unsloth and it just makes everything so much easier. I’ve been using LLMs for my programming projects and to automate my hustle but nothing at all system level or anything more than that really. I gave LTX 2.5 a go today and yeah, it works. :| Great results with text to video and unbelievable with image to video, with 20-40 second renders for a 5 second video.

The paradigm of programming is shifting too, the MCP server I showed in my last message was all done up by LLMs. It did take a little bit of nagging to get things done in a clean way that gave the LLM control and wasn’t overly prompted and all that kind of stuff. I think MCP servers may be better when written by hand, for now at least, but I got awesome results after maybe 10 hours with a custom scripting language (I didn’t want an LLM running arbitrary python, I don’t have anything virtualized and even then with all the 0-days and stuff I won’t really 100% trust future models to stay in a sandbox) and all pretty much all through prompting.

So I’ve basically only tried llama.cpp, axolotl, and unsloth, along with maybe one or two dozen programs I’ve written (i used to ask for json responses and then would process them before this mcp stuff). I haven’t played with the agentic stuff yet and don’t know where to start there. I haven’t played a 5090, 5070ti, and 96GB system RAM to play with. Still not sure whether the next upgrade is more RAM or VRAM.