I want to run a local model trained on my local files. Which model?

P

philb2

2[H]4U
Joined
May 26, 2021
Messages
3,748
So I have tens and tens of MB of Word docs,Excel Spreadsheet, Powerpoint decks, ordinaryTXT files, and PDFs on my system. I want to run a LOCAL model that I can use for searching, etc. Any ideas?
 
Last edited:
First up, what do you have for hardware to train on? Especially GPUs, but let's just go with full system specs. Or at least GPU, CPU & ram. Your hardware determines what models you can run and train.
 
zandor said:
First up, what do you have for hardware to train on? Especially GPUs, but let's just go with full system specs. Or at least GPU, CPU & ram. Your hardware determines what models you can run and train.
Click to expand...
My rig is an AMD 9900x CPU, 64 GB RAM (unfornately not maxed out at 128 GB), and an AMD 9070XT with 16 GB of VRAM.
 
Running a local AI model to query, search, and chat with your personal document collection (Word, Excel, PowerPoint, PDFs, and text files) is entirely doable on modern hardware.
The most practical approach depends on your hardware setup and whether you prefer an out-of-the-box desktop application with a graphical interface or a flexible command-line tool. Here are the best options available for a fully offline, local setup:

1. PrivateGPT or AnythingLLM (Best All-in-One Desktop Apps)​

If you want a polished graphical interface where you can just drag and drop your documents and start chatting, these are the top choices.
  • AnythingLLM: This is currently one of the most user-friendly desktop applications for local RAG (Retrieval-Augmented Generation). It runs locally, supports a wide range of document types (PDF, DOCX, XLSX, PPTX, TXT), handles the text chunking and vector embedding automatically, and lets you plug in local model runners like Ollama or LM Studio.
  • PrivateGPT: An open-source, production-ready project designed specifically for private, offline document interaction. It runs entirely on your machine and provides a local web UI.

2. Ollama + Open WebUI (Best for Power Users & Flexibility)​

If you prefer a modular setup that gives you total control over the underlying models:
  • Ollama: This runs large language models natively on Windows, macOS, and Linux with minimal setup. You can pull models like Llama 3 or Mistral directly from your terminal.
  • Open WebUI: A feature-rich, self-hosted web interface (often run via Docker or locally) that mimics the look and feel of ChatGPT. It connects seamlessly to Ollama and includes built-in Retrieval-Augmented Generation (RAG) capabilities—meaning you can upload your files directly into the chat interface to search and query them locally.

3. LM Studio (Easiest Local Model Manager)​

  • LM Studio provides a desktop application for downloading and running open-source LLMs locally (GGUF format). While its primary focus is chatting with models, recent versions feature robust local document context features, allowing you to load a folder of documents as a workspace so the model can search and reference them.

Hardware Considerations for Local RAG​

Running local models over tens of megabytes of text is very lightweight storage-wise, but the AI model itself requires decent hardware:
  • RAM/VRAM: To get fast, responsive answers, you will want a GPU with at least 8GB to 12GB of VRAM (or a system with ample unified/system RAM if running CPU/GPU hybrid inference via Ollama).
  • Embedding vs. Generation: Local RAG works by first converting your documents into "embeddings" (vector representations) stored in a local database (like Chroma or LanceDB), and then feeding only the relevant snippets to the local LLM. This means even smaller 7B or 8B parameter models can search massive document libraries accurately without needing huge hardware.
Want me to break down how to set up Ollama and AnythingLLM step-by-step on your system?
 

Recommendation: Local AI Search for Personal Documents​

For someone who has a large collection of Word documents, Excel spreadsheets, PowerPoint presentations, PDFs, and text files, I recommend building a fully local AI-powered document search system using Ollama and AnythingLLM.
This allows users to search, summarize, and ask questions about their documents without uploading sensitive files to cloud-based AI services.

Recommended software​

SoftwarePurposeCost
AnythingLLMDocument search and conversational interfaceFree
OllamaRuns AI models locallyFree
Qwen3 8BLanguage model for answering questionsFree
Qwen3 Embedding 0.6BConverts documents into searchable representationsFree
DoclingOptional advanced extraction for PDFs and Office documentsFree

What the system can do​

  • Natural-language search: Find documents by describing their contents rather than remembering exact filenames.
  • Question answering: Ask questions across multiple documents and receive AI-generated answers.
  • Summarization: Produce summaries of lengthy reports, presentations, and other documents.
  • Cross-document analysis: Compare information across multiple files.
  • Source identification: Return supporting documents and citations where the software supports them.
  • Privacy: Run document indexing, retrieval, and AI inference locally, without requiring cloud AI services.

How it works​

Document collection
Word · Excel · PowerPoint · PDF · TXT

Local document index
Text extraction, embeddings, searchable storage

Local AI model via Ollama
Retrieves relevant document passages and generates answers

AnythingLLM chat interface
Questions, summaries, and document references
This technique is called Retrieval-Augmented Generation (RAG). It does not require retraining the language model whenever documents are added.

Hardware recommendations​

HardwareSuggested specification
CPUModern 6–8 core processor
RAM32GB recommended
GPUNVIDIA GPU with 12GB+ VRAM preferred
StorageSSD with sufficient space for documents, models, and indexes
Operating systemWindows, Linux, or macOS
A dedicated GPU improves performance but isn't strictly required. Smaller models can run on CPUs, albeit more slowly.

Important considerations​

For large or frequently changing document collections, the system should support automatic folder indexing and updates.
For best retrieval accuracy, use hybrid search, combining traditional keyword search with semantic vector search. This is especially important for searching exact names, dates, invoice numbers, spreadsheet values, and technical identifiers.
Scanned PDFs require OCR. Older Office file formats may need conversion, and complex spreadsheets may require specialized extraction to preserve tables, formulas, and cell references.
For a truly offline deployment, models and dependencies should be downloaded in advance, cloud providers disabled, and local-only network configuration verified.

Final recommendation​

For most users, start with AnythingLLM Desktop + Ollama.
It's a practical, free way to create a private document-search assistant without substantial programming.
Useful links:
For organizations needing automatic indexing across extensive folder structures, accurate spreadsheet handling, and more sophisticated search controls, a custom RAG system built around Ollama, Docling, and a hybrid search database would be the better long-term solution.
 
I'd personally echo the Openwebui + Ollama + Docling for a document engine/ocr, i have that running on a t4 as kind of a personal side thing and it works fine, I generally use gemma4:12b, gemma4:e4b or granite4.2:8b . I haven't actually used the dektop piece of Openwebui as I usually run ollama/Openwebui/Docling/PGVector/ReRanker all in individual containers on a Linux host, but i'm pretty sure you can do it as an all in one thing.
 
I’ve been looking at doing almost the same thing with a pile of old documents. I wouldn’t try to feed everything into one model at once, I’d use something like Ollama with Open WebUI and let it build a local searchable library from the files. Then you can ask normal questions like “where did I mention this project?” or “show me the documents about X” instead of digging through folders manually. The nice part is that it can all stay on your own machine if you set it up that way. I’d start with a small batch of files first and see how good the search actually is before indexing everything. What hardware are you running?
 
Zepher
Thanks so much for all this information. I'm copying it to an offline txt file for reference.
maximilianx33 said:
I’ve been looking at doing almost the same thing with a pile of old documents. I wouldn’t try to feed everything into one model at once, I’d use something like Ollama with Open WebUI and let it build a local searchable library from the files. Then you can ask normal questions like “where did I mention this project?” or “show me the documents about X” instead of digging through folders manually. The nice part is that it can all stay on your own machine if you set it up that way. I’d start with a small batch of files first and see how good the search actually is before indexing everything. What hardware are you running?
Click to expand...
AMD 9900x CPU, 64 GB RAM. AMD 9070 XT GPU with 16 GB VRAM, plus NMVe and SSD drives. Fortunately bought before AI made everything unaffordable.
 
Zepher said:

2. Ollama + Open WebUI (Best for Power Users & Flexibility)​

If you prefer a modular setup that gives you total control over the underlying models:
  • Ollama: This runs large language models natively on Windows, macOS, and Linux with minimal setup. You can pull models like Llama 3 or Mistral directly from your terminal.
  • Open WebUI: A feature-rich, self-hosted web interface (often run via Docker or locally) that mimics the look and feel of ChatGPT. It connects seamlessly to Ollama and includes built-in Retrieval-Augmented Generation (RAG) capabilities—meaning you can upload your files directly into the chat interface to search and query them locally.
Click to expand...
Two reasons why not use ollama.
1. llama-cpp already does both in one package. Server inside, and WebUI outside. You install one thing, done. No docker, no configuration needed, it just works. and llama-cpp accepts every GGUF file from HuggingFace. Ollama doesn't support this format. It needs special ollama files, which leads to point no. 2:
2. https://sleepingrobots.com/dreams/stop-using-ollama/
 
Just use a RAG or second brain to hold the documents, no need to train a model. This way you can keep swapping models without having to retrain every time.
 
philb2 said:
Any chance you also have an AMD GPU? I really appreciate this info.
Click to expand...

Not currently unfortunately, I gave my 9070XT to my sister, but can help as needed or go to her place to test some stuff out as well.

Oh, actually, while not a separate card and unsure how it would work, I could always give the install a go on the steam machine I suppose.
 
mashie said:
Just use a RAG or second brain to hold the documents, no need to train a model. This way you can keep swapping models without having to retrain every time.
Click to expand...
This, specially if you want the document quoting part to be exact (and not interested into learning the document writing style or excel technic per say), not some inference and adding document later on is much easier than redoing a new fine-tuning run every time.
 
You must log in or register to reply here.
Back
Top