Large language models are impressive, but out of the box they know nothing about your products, policies, contracts or customers — and they can confidently invent answers. Retrieval-augmented generation (RAG) is the most common way to fix that. This guide explains what RAG is, how it works and what it takes to build one well.
RAG in one paragraph
When a user asks a question, a RAG system first retrieves the most relevant passages from your own content — documents, knowledge base articles, database records — and then asks the language model to generate an answer using only those passages. The model becomes a skilled reader of your information rather than a source of facts on its own.
How RAG works, step by step
Ingestion (done in advance)
- Collect the content: PDFs, Word files, web pages, tickets, database rows.
- Clean and split it into chunks — for example sections of a few paragraphs — with metadata such as title, date, department and access permissions.
- Embed each chunk: an embedding model turns text into a vector (a list of numbers) that represents its meaning.
- Store the vectors in a vector index — a dedicated vector database, or vector support in PostgreSQL (pgvector) or SQL Server / Azure SQL.
Answering (at query time)
- Embed the user's question.
- Find the most similar chunks, filtered by what that user is allowed to see.
- Build a prompt: instructions, the retrieved chunks and the question.
- Call the language model and return the answer with citations to the source documents.
Where RAG helps businesses
- Internal knowledge assistants for policies, procedures and technical documentation.
- Customer support answers drawn from help-center content, with hand-off to humans.
- Sales and pre-sales assistants that answer product and compliance questions.
- Search inside products — for example searching contracts, reports or case notes by meaning, not just keywords.
RAG vs fine-tuning
| RAG | Fine-tuning | |
|---|---|---|
| Best for | Answering from changing company knowledge | Teaching a style, format or narrow task |
| Keeping information current | Re-index documents | Retrain the model |
| Citations | Natural | Difficult |
| Access control | Filter at retrieval time | Not practical |
What makes a RAG system good
- Chunking that respects structure: split by headings and sections, not arbitrary character counts.
- Hybrid search: combine vector search with keyword search so exact terms like product codes are not missed.
- Permission-aware retrieval: filter chunks by the user's access rights before they reach the model.
- Grounded prompts: instruct the model to answer only from the provided sources and to say when the answer is not there.
- Evaluation: a test set of real questions with expected answers, run whenever you change prompts, models or chunking.
- Freshness: re-index automatically when source content changes.
Common pitfalls
- Indexing everything, including outdated or contradictory documents.
- Ignoring permissions, so users can see content they should not.
- No citations, which makes answers impossible to trust or verify.
- No evaluation, so quality silently drops after changes.
- Sending more data to the model provider than necessary.
What RAG costs to run
Running costs come from embedding documents (mostly one-off, plus updates), storing vectors, and model calls per question — which grow with the amount of retrieved text you include. Limiting chunks to the most relevant few, caching frequent answers and choosing the right model for the task keep costs predictable. Our LLM integration checklist covers cost control in more detail.
Key takeaways: RAG grounds AI answers in your own content by retrieving relevant passages before generating. Success depends on good chunking, hybrid and permission-aware search, citations and continuous evaluation.
Want to add AI search or an assistant to your product? See our AI integration services, including RAG development on .NET and Node.js, or read how to integrate an LLM API into ASP.NET Core.
