An AI assistant on your own content, without the hype

What RAG actually is

If you want an AI assistant that answers questions about your business using your own documents, the technique you are after is usually called retrieval-augmented generation, or RAG. The idea is simpler than the acronym. When someone asks a question, the system first retrieves the most relevant passages from your content, then hands those passages to the language model and asks it to write an answer grounded in that material instead of in whatever it absorbed during training. Ingest and index your content, retrieve the relevant chunks for a query, add them to the prompt, then generate. That is the whole loop.

It helps to contrast RAG with the two things people often reach for instead. The first is a raw model with no access to your content. It will answer fluently, but it is drawing on generic training data, so it can confidently invent specifics about your pricing, your policies, or your products. The second is a scripted, decision-tree chatbot. It only handles the questions someone wrote in advance, so it falls apart the moment a customer phrases things differently. RAG sits in between: it lets the model reason and phrase naturally, but it tethers the answer to passages pulled from your actual material.

The parts that make it work

A working assistant is a chain of concrete components, and each one can go right or wrong. It starts with clean source content, because everything downstream inherits its quality. Documents are then split into smaller chunks, since a full document is too large for the model's context and too coarse for precise retrieval. Chunk size is a real tradeoff: larger chunks carry more surrounding context but drag in irrelevant text, while smaller chunks are more precise but can lose the thread. Each chunk is turned into an embedding, which is a list of numbers representing the meaning of the text, so that passages can be matched by meaning rather than exact keywords. Those embeddings live in a vector or search index. At query time the system retrieves the top handful of most similar chunks, the model writes the answer from them, and the answer carries citations back to the source passages. Around all of that sit guardrails and a path to hand off to a human.

Citations and the human handoff are not afterthoughts. They are the trust mechanism. Citations let a user verify the answer against the original passage instead of taking the assistant's word for it. A clean escalation path means that when the assistant is unsure, or the topic is sensitive, or the customer simply asks for a person, there is somewhere safe for the conversation to go. A grounded assistant should escalate when it lacks enough grounded information to answer safely, when a user asks for a human, or on hard-coded sensitive topics like legal, medical, or financial disputes, and it should carry the conversation context along on the way out.

It reduces hallucination. It does not eliminate it.

This is the part where most marketing goes wrong, so we want to be direct. Grounding the model on your own content meaningfully reduces errors, and the effect can be large. In a controlled study of cancer-information chatbots, hallucination rates were around 40 percent with no external references, dropped to somewhere between 6 and 35 percent when grounded on general web search, and fell to between 0 and 6 percent when grounded on a curated, domain-specific knowledge base. The lesson is that clean, relevant source content drives accuracy far more than the choice of model. But notice the floor is not reliably zero.

You do not have to take a single study's word for it. A preregistered Stanford study of commercial, RAG-based legal research tools found they still hallucinated between roughly 17 and 33 percent of the time, and the authors concluded plainly that vendor claims of eliminating hallucinations through RAG are overstated. So when someone promises you a hallucination-free assistant, that promise is contradicted by the best available evidence. The honest version is this: a well-built assistant will be right far more often than a raw model, and it will still be wrong sometimes, which is exactly why citations and a human fallback matter.

Retrieval is where many wrong answers are born

When a grounded assistant gives a bad answer, the model is often not the culprit. The retrieval step failed to surface the right passage, so the model never had the material it needed. The encouraging part is that retrieval is a tunable, measurable engineering problem rather than a mystery. In Anthropic's experiments, a standard retrieval baseline missed the relevant chunk in its top results about 5.7 percent of the time, and layering on better techniques like contextual embeddings, lexical matching, and a reranking step brought that down to roughly 1.9 percent. The exact numbers will vary on your content, but the shape holds: you can measure retrieval, and you can improve it. What you cannot do is fix it by improving the model. If the source content is wrong or missing, the answer will be too.

You might not even need RAG

Here is the kind of thing a studio that does not overpromise will tell you up front. If your knowledge base is small, you may not need a retrieval pipeline at all. Anthropic notes that for a knowledge base under roughly 200,000 tokens, on the order of 500 pages, you can simply place the entire body of content in the prompt rather than building and maintaining retrieval infrastructure. Full RAG earns its keep once the content grows past what fits comfortably in context. Worth asking before anyone quotes you a build for machinery you do not yet need.

Data and privacy, handled deliberately

An assistant grounded on your content can only be as careful as you make it about what that content includes. Be deliberate about which documents it is allowed to retrieve from. Enforce per-user permissions at retrieval time, so that every passage returned is checked against what the current user is actually allowed to see, rather than assuming the index itself is the boundary. And redact or tokenize sensitive personal or health information before it is ever embedded, because embeddings do not reliably obscure the underlying text and the vector store can leak through misconfiguration or adversarial attacks. These are not exotic precautions. They are the baseline for putting any real business content behind an assistant.

What it costs, in plain buckets

We will not quote you a per-query dollar figure, because the honest answer is that it depends on your content and traffic, and any number we invented would age badly. It is more useful to think in three buckets. There is a one-time build: preparing and chunking content, generating embeddings, and wiring up retrieval, citations, and guardrails. There is an ongoing per-query cost: each question triggers a model call plus a retrieval step. And there is upkeep: re-indexing as your content changes, monitoring answer quality, and fixing the answers that come back wrong. The upkeep bucket is the one people forget. An assistant grounded on content that has gone stale will confidently cite the old version.

How to actually evaluate one

Do not accept a demo of five cherry-picked questions as proof. Build a test set of real questions with known correct answers, drawn from what your customers and staff actually ask. Then measure three things. First, answer accuracy against those known answers. Second, citation correctness: does the passage the assistant cites actually support the claim it made, or is it pointing at something unrelated? Third, refusal and handoff behavior: when the answer genuinely is not in your content, does the assistant correctly decline and route to a human, or does it guess? That third measure matters because grounded systems stay accurate partly by being willing to say they do not know. In the cancer-chatbot study, the grounded systems answered a smaller share of questions but were far more accurate on the ones they did answer, while the ungrounded bots answered everything and were wrong much more often.

None of this requires you to become an expert in embeddings. It requires you to hold a builder to the same standard you would hold any other vendor: show me it works on my real questions, show me where it points for each answer, and show me what it does when it does not know. An assistant on your own content is a genuinely useful thing to build. It is just not a magic one, and anyone who tells you otherwise is selling the hype, not the system.

Sources

External sources are provided for verification. NavoTech is not affiliated with and does not endorse the organizations cited.

Written by the team at NavoTech Digital Solutions. Have a project or counter-example? Get in touch.