How to Build a RAG Chatbot for Your Company Documents (Without It Making Things Up)

Many companies want an assistant that can answer questions from their own material: policies, product manuals, support articles, contracts or past project notes. The standard approach is retrieval-augmented generation (RAG). The system finds the most relevant passages in your documents and gives them to a language model, which writes an answer based on them. If you are unsure whether RAG is the right choice compared with fine-tuning, read RAG vs fine-tuning first. This article assumes you have decided on RAG and want to build it well.

The pipeline in one picture

1. Ingest: collect documents from where they live and convert them to clean text.
2. Split: cut the text into passages (chunks) that are small enough to retrieve precisely.
3. Index: store each chunk with a numeric representation (embedding) and its metadata.
4. Retrieve: for each question, find the best matching chunks.
5. Generate: ask the model to answer using only those chunks, and to cite them.
6. Evaluate: measure quality on real questions, continuously.

Step 1: Get the documents clean

Most poor RAG systems fail here, not in the model. Scanned PDFs without text, tables flattened into nonsense, headers and footers repeated on every page, and several outdated versions of the same policy will all lead to wrong answers. Extract text carefully, keep headings, remove repeated boilerplate, and record where each document came from, who owns it and when it was last updated. Decide which version of a document is the source of truth.

Step 2: Chunk with structure, not just size

Splitting every document into fixed-size blocks is easy and often mediocre. Better chunks follow the document's structure: a section under a heading, a question and its answer, a table row group. Add a small overlap so ideas that cross a boundary are not lost, and attach the document title and section heading to each chunk so it makes sense on its own. There is no universal best size; choose by testing on your own questions.

Step 3: Retrieve with more than one signal

Embedding search finds passages that are similar in meaning, which is great for paraphrased questions. It can miss exact terms such as product codes, names and clause numbers. Keyword search has the opposite strengths. A common and effective approach is hybrid retrieval: run both, merge the results, then optionally re-rank the top candidates with a model that scores how well each passage answers the question. Retrieve a handful of strong passages rather than a huge pile; irrelevant context makes answers worse.

Step 4: Respect permissions

This is the point that causes real incidents. If a junior employee can ask the chatbot about salaries because the documents were indexed together, you have built a data leak. Store access information with every chunk and filter retrieval by what the signed-in user is allowed to see, before anything reaches the model. Do not rely on telling the model to keep secrets.

Step 5: Make the model answer from the sources

Instruct the model to use only the supplied passages, to say clearly when they do not contain the answer, and to cite the document and section for each claim. Show the citations in the interface as links so a person can verify them in one click. Trust grows when users can check, and a respectful 'I could not find this in the documents' is far better than a confident guess.

A useful rule: if the system cannot cite a source, it should not state the claim as fact. Build the interface so that missing citations are visible.

Step 6: Test it before you trust it

Collect fifty to a couple of hundred real questions from the people who will use the system, and write down what a correct answer looks like and which document contains it. Run these every time you change chunking, retrieval, the prompt or the model. Look at two things separately: did retrieval bring back the right passage, and did the answer use it correctly? Many teams discover that retrieval, not the model, is the weak point. Include questions that should be refused because the answer is not in the documents.

Step 7: Plan the unglamorous parts

• Freshness: how do new and changed documents reach the index, and how fast? Deleted documents must disappear too.
• Privacy and hosting: decide where documents and questions are processed, and read your provider's data handling terms for the plan you use.
• Cost and speed: each question involves retrieval and a model call. Measure both, and cache where it is safe.
• Feedback: add a simple thumbs up or down with a comment, and review the bad answers every week.
• Ownership: someone must be responsible for the content. A chatbot cannot fix documents that are wrong.

Common mistakes

• Indexing everything, including drafts and obsolete files.
• Skipping evaluation and judging by a few impressive demos.
• Putting permissions in the prompt instead of in retrieval.
• Using one chatbot for every purpose instead of starting with one well-defined use case, such as internal support or HR policy.

Start small

The best first project is a single, well-bounded collection of documents with a clear group of users and a measurable goal, such as fewer repeated questions to a support team. We build RAG systems and autonomous agent workflows for production use, with evaluation and access control designed in from the start. If you would like to scope one, get in touch. For help estimating the budget of any build, see our software cost guide.