A knowledge base that cites its sources

The short answer: a RAG knowledge base earns trust by being checkable, not clever. (RAG, short for retrieval-augmented generation, means the assistant looks up the right passages in your own documents first, then writes an answer from them.) Every answer shows which document and section it came from, the system refuses to answer when it finds nothing good, documents are split along their own structure, search combines exact words with meaning, and the library updates whenever a source changes. The goal is never to be confidently wrong.

Why most internal chatbots get abandoned

They get built. They demo well. Six weeks later nobody is using them.

The reason is almost always the same, and it is not the quality of the AI model. Somebody asked it something that mattered, got a confident answer, acted on it, and found out later it was invented. They told two colleagues. After that, nobody trusts it, and a tool nobody trusts is worse than no tool: it is dead weight that somebody is still paying for.

The imbalance is brutal. A hundred correct answers build trust slowly. One confident wrong answer on something that matters destroys it immediately, because the mistake was invisible at the moment it happened.

So the design goal for an internal knowledge base is not "answer as many questions as possible". It is never be confidently wrong, and everything below follows from accepting that trade.

Citations change what the tool is

When every answer links back to the document and the section it came from, the assistant stops being an oracle and becomes a very fast search of your own files.

That is much easier to trust, and a more honest description of what the tool does. It also changes how people behave, in exactly the right way: on anything important, they click through and check.

In practice that means two rules:

• The assistant may only answer from the passages it found. Not "prefer" them. Only.
• Every claim needs a source, and an answer that cannot point to one does not get shown.

A cited answer that turns out to be wrong is a recoverable mistake: the person opens the source, sees the mismatch, and reports an out-of-date document. An uncited wrong answer is unrecoverable, because nobody can tell where it came from.

Build the "I don't know" into the system, not the instructions

The most important behaviour is what happens when the search finds nothing good.

The right answer is to say nothing. "I could not find this in the indexed documents" is a useful reply. It tells the person to ask a human, which is what they would have done anyway, and it costs them ten seconds, not a wrong decision.

The mistake is asking the AI model to do this. Instructions like "only answer if you are confident, otherwise say you do not know" work most of the time, which is the worst possible quality for a safety feature. A model that wants to be helpful will fill a silence with something plausible, and it will do it on the obscure questions where the search failed, which are exactly the ones where being wrong is most likely.

So the limit belongs in the system itself. Score what the search found. If the best match is below the bar, the AI is never asked to write an answer at all. It is not told to decline. It is simply not called.

This is the same idea as keeping arithmetic out of an AI model. An AI model never decides to stop. It always produces output. If you need it to stop, you have to build the stop around it.

Splitting documents is most of your quality

The choice of AI model gets the attention. How you split your documents decides the outcome.

The system does not read whole documents. It cuts them into pieces (often called chunks) and searches those. Cut too small and you find a fragment with no context: a sentence saying "this does not apply to contractors" with nothing to show what "this" was. Cut too large and one piece covers several topics, so it matches nothing well.

What works better than tuning a number of characters:

• Split along the document's own structure, not at a fixed length. Headings, sections, list boundaries. A piece that matches a section is a piece that answers a question.
• Overlap the pieces slightly so a point that crosses a boundary survives in one piece.
• Label every piece: document title, section heading, last-edited date and source link. This is not optional bookkeeping. It is what makes a citation possible at all. A piece with no label can be found but never cited, which means it can never be trusted.

Search by meaning fails on the words people actually type

Search by meaning (the technical name is vector search) is excellent at "what is our policy on late invoices?" and surprisingly bad at INV-2024-0847.

Exact references such as invoice numbers, product codes, error codes, client names and legal references are exactly where meaning-based search struggles, because its whole point is to look past the exact wording to the idea behind it, and a reference number has no idea behind it. It is exact wording and nothing else.

Hybrid search fixes this: run an exact-word search and a meaning search together, and combine the results. The exact-word search catches the precise reference, the meaning search catches the paraphrase, and together they cover what people really ask. In an operations knowledge base, where half the questions contain a reference number, this is not a refinement. It is the difference between a system that works and one that fails on the questions people care most about.

A library built once is wrong within a month

Documents change. Policies get revised. Prices move. Somebody edits the onboarding guide and does not tell anyone.

If the library is built once, you eventually get the worst outcome available: a confident, correctly cited answer pointing at a version that no longer exists. Now the citation works against you, because it lends authority to something out of date.

So the watcher on the source folder matters more than the choice of AI model. When a document changes, its pieces are rebuilt and, critically, the old pieces are deleted. Rebuilding without deleting leaves both versions competing to answer the same question, and which one wins is close to random.

Two details worth building in from the start:

• Stamp each piece with the source's last-edited date and show it in the answer. "From the Onboarding Guide, updated 12 March" lets a person judge for themselves whether it is current.
• Handle deletions. A document removed from the folder must have its pieces removed too, or you will be citing a file that no longer exists.

Let the assistant ask live systems, not read a copy

Not everything belongs in the library, and it is worth being deliberate about where the line is.

Anything that changes by the minute (availability, order status, stock levels, ticket state) should be looked up live, not stored as a piece of text. Storing live information guarantees the library is wrong, and dressing that wrongness in a citation makes it worse.

The assistant on this site works exactly that way. When someone asks what times are free, it does not read a table of bookings. It asks the same availability check the booking calendar uses, through the same protected connection, so the assistant and the calendar cannot disagree. There is one source of truth and two readers of it.

That is the general pattern: search for what is written down, ask a live system for what is true right now.

One detail that catches people: each visitor's conversation needs its own ID that stays the same during their visit and is different from everyone else's. Get that wrong and every visitor shares one memory, which is both a privacy problem and a very confusing bug to track down.

The part that is not about AI at all

Most of the work in a knowledge base that works is deciding what goes into it.

Putting everything in produces a system that finds plausible-looking noise. Every old draft, every meeting note and every half-finished proposal becomes a candidate answer with the same standing as the real policy.

Choose the documents that answer real, repeated questions. The way to find them is not to guess: look at what people actually ask each other in Slack, and add the documents that answer those questions. Old drafts, personal notes and anything replaced should be left out, or clearly marked so the search can filter them.

That is a librarian's job, not an engineer's, and it is where most of the value comes from. A small, curated, current library beats a large, complete, out-of-date one every time.

How I know it works

Gut feeling does not survive contact with a real user, so before handover I build a test set: thirty to fifty real questions drawn from what the team actually asks, each with a known correct answer and a known correct source.

That set gets run before and after every change, such as a new way of splitting documents, a different AI model or a rewritten instruction. Without it you are making changes and hoping, and these systems are good at looking like they improved while quietly getting worse on the unusual questions.

Two things I measure beyond plain accuracy:

• How often it correctly refuses. A system that never says "I do not know" is not confident. It is wrong on the questions you have not checked. I deliberately include questions the library cannot answer, and the correct result is a refusal.
• Whether the source is right, separately from whether the answer is right. An answer can be correct while pointing at the wrong document, and that is a trust problem waiting to happen the first time someone clicks through.

The short version

An internal knowledge base earns its place by being checkable, not by being clever.

Cite everything. Build the refusal into the system instead of asking the model nicely. Split documents along their structure and label each piece. Search by exact words and by meaning together, because people type reference numbers. Update the library when a source changes and delete the old pieces. Ask live systems for live facts. Choose what goes in with care.

Do that and you get something people keep using after the novelty wears off, which is the only measure of an internal tool that matters.

The refusal and replay ideas here are the same ones behind error handling in n8n workflows. If you want a knowledge base or document-reading workflow built and handed over with its test set, see my Claude and OpenAI document and knowledge-base service.

Frequently asked questions

What is RAG, in plain English?

RAG stands for retrieval-augmented generation. The assistant first looks up the relevant passages in your own documents, then writes an answer using only what it found. The AI model supplies the wording and your documents supply the facts, which makes the answer traceable to a source, something a model answering from memory alone cannot offer.

How do you stop a RAG chatbot from making things up?

Do not rely on the instructions you give the AI. Score the passages the search found and, if the best one is below a set bar, never ask the AI to answer at all. Return "I could not find this in the indexed documents" instead. Then require a source for every claim, so an answer that cannot cite anything is not shown.

What is hybrid search in RAG?

Hybrid search runs an exact-word search and a meaning-based search together and combines the results. Meaning-based search finds rephrased versions of a question, and exact-word search catches precise references such as invoice numbers, product codes and error codes, which meaning-based search handles poorly. Using both covers the way people actually ask.

How do you test whether a knowledge base works?

Build a test set of thirty to fifty real questions, each with a known correct answer and source, and run it before and after every change. Check that the source is correct separately from the answer, and include questions the library cannot answer, where the correct result is a refusal.

How often should a RAG knowledge base be updated?

Whenever a source document changes. Rebuild the changed pieces, delete the old ones so two versions never compete to answer the same question, and remove the pieces of any deleted file. A library built once is wrong within a month.