The Minimum Viable RAG: What a Small Team Actually Needs

Most small companies now have the same want. They have a pile of documents, contracts, policies, past tickets, internal wikis, product manuals, and they want to ask questions of that pile in plain English and get answers back. The general-purpose chatbots cannot do this. They do not know your documents. They were trained on the public internet, not on your renewal terms or your onboarding runbook.

The assumption is that closing this gap takes a machine-learning hire, a graphics card, and a large monthly bill. It does not. The technique that makes it work is well understood, most of it runs on an ordinary server, and the part that decides whether the result is good or useless is not the part anyone expects.

This is a walkthrough of that technique for a technical reader who does not do machine learning. Four moving parts, in plain language, and where the real work actually sits.

The problem

A model answers from what it learned during training. Ask it something specific to your business and it does one of two things. It says it does not know, or it invents a plausible answer. The second failure is the dangerous one, because a confident wrong answer looks exactly like a right one.

You cannot fix this by training your own model. Training is expensive, slow, and out of reach for a small team. And even if you could, the moment a document changes you would be stale again. What you want is a system that reads your current documents at the moment you ask the question, and answers from what it actually read.

The analysis

That system has a name. It is retrieval-augmented generation, or RAG. The idea comes from a 2020 paper by Lewis and colleagues at what was then Facebook AI (Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks). You pair a retriever with a generator. The retriever finds the passages in your documents that relate to the question. The generator, the language model, writes an answer constrained to those passages.

The reason this matters beyond convenience is trust. Because the model is reading passages you handed it rather than recalling from training, it can point at the exact source of every claim, and it is far less likely to make something up. The same paper that introduced the method is the source for that property: retrieval conditions the generation. Ground the answer in real text, and you can cite the text.

The pipeline has four parts. Understanding each one tells you where the cost is and where the difficulty is, and they are not the same place.

The solution

Part one: ingest and chunk. You take your documents and cut them into pieces. Not one giant blob, and not single sentences, but passages of a few paragraphs each. Every piece has to stand on its own, because at query time the model will see only the pieces that got retrieved, not the whole document. This is the ingest-and-chunk step, and it is where a RAG system is quietly won or lost. More on that below.

Part two: embed. Each chunk gets converted into a vector, a list of numbers that captures its meaning. Two passages about the same topic land near each other in this number space even if they use different words. This is what lets a search for “how long to appeal a denial” find a paragraph that says “grievances must be filed within 180 days.” Open-source embedding models do this, they are free to use, and they run on an ordinary CPU with no graphics card at all (sentence-transformers). The retrieval half of RAG needs no GPU.

Part three: store. The vectors go into an index that can find nearest neighbors quickly. You do not need a specialized vector database for this. The pgvector extension adds vector similarity search to plain PostgreSQL (pgvector), so one Postgres box you may already run can be both your normal data store and your vector index. The whole embed, store, retrieve flow is standard and documented (sentence-transformers semantic search).

Part four: retrieve and generate. At query time you embed the incoming question the same way you embedded the chunks, search the index for the closest passages, and hand those passages to the language model as context along with the question. The model writes the answer from that context. This last step is the only part that needs a capable language model, and even that can point at a model seat your team already pays for rather than a new subscription.

If you want to see the four parts assembled into something that actually answers questions, we walked through a working build in Building a Privacy-First Healthcare RAG Assistant That Runs Entirely on Local Hardware. Same four parts, running end to end on one machine.

One more piece is worth knowing about, because it changes the economics. The Model Context Protocol, or MCP, is an open standard for connecting AI applications to external tools and data (Model Context Protocol). Expose your document index through MCP, and any MCP-compatible client can query it without custom glue written for each client. Your index becomes a thing you own and plug models into, rather than a thing welded to one vendor.

Lessons

Three things fall out of this that are worth holding onto.

The compute is cheap and mostly CPU-bound. The retriever, the embeddings, the vector search, all of it runs on a normal server. There is no GPU in the retrieval path and no monthly floor to clear before it does anything useful. The only step that reaches for a heavier model is the final answer-writing, and that is a small fraction of the work. The mental model where AI means renting expensive hardware does not apply to most of this pipeline.

Document preparation is the hard part, not the math. The embedding models are free and the vector search is a solved problem. What decides whether your system returns the right passage or the wrong one is how well you chunked messy real-world files. PDFs with two-column layouts, scanned pages, spreadsheets, tables that mean nothing once flattened into a line of text. If a chunk arrives garbled or split down the middle of the sentence that mattered, no amount of compute recovers it. This is the same lesson from a different angle as The Belief Gap vs. the Readiness Gap in Healthcare AI: the AI project is usually a data project wearing a costume. The model is the easy 20 percent. Getting clean, well-formed source text into it is the other 80.

A general-purpose chatbot has the opposite problem from a document one: it has read enormous amounts of the public internet, so it sounds fluent and confident on almost anything, but it has never seen your renewal terms, your onboarding runbook, or the thing your team decided in last Tuesday’s meeting. It is current on the world and stale on you. This shows up as a very ordinary, very expensive kind of friction. A specific figure or decision exists somewhere in the company, but no single person knows where, so the search for it becomes a search for a person instead of a document: five different conversations to find out who owns the dashboard that has it, even with a data-catalog tool already in place to prevent exactly that. Every one of those conversations is a real cost, paid over and over by different people asking the same kind of question. A working retrieval index does not eliminate the need for people who understand the business. It does mean the answer to “where is that number” stops depending on whoever happens to remember.

The durable advantage is owning your data layer, and it stays a job, not a one-time build. When you build RAG this way, the corpus stays yours. It lives on your box, in your Postgres, in your index. The only thing that ever leaves is the question plus a handful of retrieved snippets, sent to whatever model writes the answer. That model is swappable. You can point at a local one, or a cloud seat you already have, or switch when a better one ships next quarter, without moving your data or rebuilding anything. The data layer is the asset. The model is a commodity you rent by the question.

The part that is easy to underestimate going in is that the index is never finished. New documents arrive, old ones change, a policy gets revised and the chunk built from the old version is now confidently wrong instead of just missing. Someone has to own re-ingesting what changed, catching the chunks that broke, and keeping the index honest against the current state of the business, on an ongoing basis, not as a one-time setup task. That is a real, recurring role, closer to data quality work than to software development, and it is the piece most small teams do not staff for because it was never obvious it existed until the index started drifting.

None of this requires an ML hire, a GPU, or a large subscription. It requires taking your own documents seriously enough to prepare them well, and keeping that work current after launch. That is the whole trick, and it has been hiding in plain sight since 2020.