![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
Uncensored A.I. chat software that you can run locally on your PC
- Thread starter ZodaEX
- Start date
grim4593
Limp Gawd
- Joined
- Nov 30, 2014
- Messages
- 444
It depends on your hardware and what you are looking to get out of it. There is quite a bit of discussion here: https://www.reddit.com/r/LocalLLaMA/
Coding assistants are popular, role playing chats, and there are now options to read local files as a backend dataset.
The largest factor right now is video card RAM (bonus points if you have a newer Apple M2 device since it has unified video memory).
The more GPU memory you have the faster the LLMs can process and the larger (e.g. smarter) the models you can use.
I use this single executable program to run models locally: https://github.com/LostRuins/koboldcpp
A typical model is anywhere between 8-40GB.
Nvidia just dropped this but no idea how good it is: https://www.nvidia.com/en-us/ai-on-rtx/chat-with-rtx-generative-ai/
It looks like it is based off Mistral 7B which is censored but I would be surprised if people get it to run other models too.
Coding assistants are popular, role playing chats, and there are now options to read local files as a backend dataset.
The largest factor right now is video card RAM (bonus points if you have a newer Apple M2 device since it has unified video memory).
The more GPU memory you have the faster the LLMs can process and the larger (e.g. smarter) the models you can use.
I use this single executable program to run models locally: https://github.com/LostRuins/koboldcpp
A typical model is anywhere between 8-40GB.
Nvidia just dropped this but no idea how good it is: https://www.nvidia.com/en-us/ai-on-rtx/chat-with-rtx-generative-ai/
It looks like it is based off Mistral 7B which is censored but I would be surprised if people get it to run other models too.
Just tried it, it can read your local doc, txt, pdf and is giving pretty good answer, but it seem my 3070 8GB vram is close to the limit and was not impressed by the speed of response at all on my system (not faster than bingGPT at all, even slower, make sense compute is much bigger for this than the little text to send on fast Internet), it was using almost 100% of the gpu resource.
Last edited:
grim4593
Limp Gawd
- Joined
- Nov 30, 2014
- Messages
- 444
I have a 3070TI and things ran decent enough. I threw in some old DND 5E books and it was able to pull up items from stat blocks and tables decently enough. The non-OCR book was a bit of a struggle. Might be a good lookup helper for a DM.
I also have 64GB of ram and it is a matter of trading speed for quality. I can technically run a 70B but it becomes an exercise of patience.
I also have 64GB of ram and it is a matter of trading speed for quality. I can technically run a 70B but it becomes an exercise of patience.
This is a good place for uncensored AI chat: https://flowgpt.com/
I know you're wanting something local, but there's still some useful things on here.
I know you're wanting something local, but there's still some useful things on here.
sharknice
2[H]4U
- Joined
- Nov 12, 2012
- Messages
- 3,759
From the looks of it I think the new NVIDIA thing is basically just an interface they made for the same models you would get from hugging face. I haven't actually tried it though. I've used this https://github.com/oobabooga/text-generation-webui which works pretty well. They can work pretty well depending on the models you download. I only really just fool around with them a bit. I never load documents and use it as a real assistant.
If you want any privacy at all running them on a machine at home will be the way to go in the future as they become more useful. There are already companies building machines just to run AI at home.
If you want any privacy at all running them on a machine at home will be the way to go in the future as they become more useful. There are already companies building machines just to run AI at home.