Building a Privacy-First Healthcare RAG Assistant That Runs Entirely on Local Hardware

Ask a health plan a simple question, like “Do I need prior authorization for an MRI?”, and the answer is almost always sitting in a document somewhere. The problem is that it is one paragraph buried across dozens of benefit summaries, prior-authorization policies, formularies, and appeals procedures. Finding it means knowing which document to open and then reading the whole thing.

I built a small system that answers those questions directly, grounds every answer in the actual plan documents, cites its sources, and refuses to answer when the documents do not cover the question. The part I care about most: it runs entirely on local hardware. No member data ever leaves the machine.

This post is the writeup of that project. It is also the first entry in a longer idea I am building toward, which I explain at the end.

The problem

Members and the service representatives who support them ask the same kinds of questions all day. How much is my specialist copay? How long do I have to file an appeal? My claim was denied for no prior authorization, what can I do?

A general-purpose chatbot cannot help with these. It does not know this specific plan’s rules, and in healthcare you cannot paste member or policy data into a cloud model. That is a compliance non-starter. So the useful answer to “can AI help here” has two hard requirements attached to it.

The analysis

The right tool is retrieval-augmented generation, or RAG. Instead of asking a model to answer from memory, you first retrieve the exact passages that address the question, then ask the model to compose an answer that is constrained to those passages, with citations. Two requirements shaped every decision:

1. Grounding and trust. In healthcare, a confidently wrong answer is worse than no answer at all. The system has to cite the source of every claim, and it has to decline when the documents do not contain the answer, rather than fill the gap with something plausible.
2. Privacy by architecture, not by policy. The only durable way to satisfy healthcare privacy is to never move the data off the machine in the first place. That means local embeddings and a local model, with a cloud model available only as an explicit opt-in.

The solution

The whole pipeline is small and boring, which is the point.

   member question
        │
        ▼
   embed the question        bge-m3, served locally by Ollama
        │
        ▼
   vector search             Postgres + pgvector, exact cosine search
        │
        ▼
   compose the answer        qwen2.5, served locally by Ollama
   (grounded, cited,         refuses when retrieval comes up short
    constrained to passages)
        │
        ▼
   answer + cited sources

Every step runs on the host. Nothing is sent out.

• Embeddings. The bge-m3 model produces a 1024-dimension vector for each chunk of text and for each incoming question. It is served locally by Ollama.
• Vector store. PostgreSQL with the pgvector extension holds the chunk vectors and does the similarity search.
• Generation. By default a local qwen2.5 model writes the answer. There are no API keys and no network calls. A cloud model can be swapped in with a single environment variable when higher answer quality is worth it, but the default path needs nothing but the local box.
• Guardrails. The system prompt forbids outside knowledge, requires inline bracketed citations like [1] and [2], and instructs the model to say it does not have the information when retrieval does not support an answer. The code path also short-circuits to a refusal when nothing relevant is retrieved.

The interface is a single page. You type a question, you get an answer with inline citation markers, and you can expand each cited source to read the exact passage it came from, along with its similarity score. That transparency is deliberate. A member, or a compliance reviewer, can always see why the system said what it said.

The outcome

Over a synthetic health-plan knowledge base, the assistant does what it should:

Question	Answer	Cited source
Do I need prior auth for an MRI?	Yes. Advanced imaging requires prior authorization; standard determinations are made within 15 calendar days.	Prior Authorization Policy
How long do I have to file an appeal?	180 calendar days from the adverse determination notice.	Appeals and Grievances
What is my specialist copay?	50 dollars in-network, 40 percent after deductible out-of-network.	Summary of Benefits
What color is the sky?	”I do not have that information in the plan documents.” (refuses)	none

It answers correctly, it cites the specific policy behind each answer, and it declines the out-of-scope question instead of inventing something. And it does all of that with embeddings, retrieval, and generation running locally, with no cloud dependency.

Every document in the knowledge base is synthetic. There is no real member data, no protected health information, and no real payer anywhere in the project. In a real deployment, that synthetic corpus is simply replaced by the institution’s own private documents, which stay on the host exactly as the synthetic ones do.

What I learned

The most useful lesson came from a bug. My first version created an approximate vector index (IVFFlat) up front, the kind of thing you reach for to make search fast at scale. On my small corpus it quietly returned the wrong documents. For a question about MRI prior authorization, it surfaced the enrollment policy and never even considered the prior-authorization document.

The reason is that an approximate index partitions vectors into clusters and, by default, only searches the nearest one. With a handful of vectors spread across a hundred clusters, most clusters are empty and the search misses the passages that actually matter. The fix was to stop optimizing for a scale I did not have. At this size, an exact search does a full scan in milliseconds and gives perfect recall. The “scale” optimization was making the system both slower to reason about and less accurate. I left a note in the code marking exactly where an approximate index should be turned back on once the corpus is large enough to need it.

That is the whole lesson: measure before you optimize, and be honest about the scale you are actually operating at.

Two other decisions held up well. Making the generation provider a single swappable seam meant “local by default, cloud by choice” was a one-line change rather than a rewrite. And treating refusal as a feature, not a failure, is what makes the system trustworthy. A benefits assistant that declines to guess is one you can actually put in front of a member.

Why this is only the first node

I think of this project as Node Zero. The larger idea is a distributed, privacy-first healthcare AI, where each institution self-hosts its own node, keeps its own data on its own premises, and can optionally share GPU compute for heavier models without ever sharing the underlying data. The hard constraint of that vision, that sensitive data never leaves the building, is exactly the design constraint of this single node today.

The ideas I want to build on top of this, in roughly the order I plan to tackle them:

• A reranking pass to tighten retrieval precision before the answer is composed.
• Per-document access control and audit logging, so a node can prove who saw what.
• Multi-node federation, where separate institutions each run a node and pool compute rather than data.
• Streaming answers and short-term conversation memory for follow-up questions.

I will write each of those up here as I build them. If you are working on anything in this space, or you just want to compare notes, reach out.