What is RAG architecture?

RAG architecture is built as a two-phase pipeline: an offline step that indexes your documents into a searchable form, and a runtime step that retrieves the right pieces and hands them to the model before it answers.

What happens at each step of the RAG pipeline?

Here's what actually happens under the hood, in order:

1. Chunking and indexing (offline): Your documents — PDFs, wiki pages, support tickets, whatever — get split into small chunks and converted into embeddings (number lists that capture meaning), then stored in a vector database.
2. Query embedding: When a user asks a question, that question gets converted into an embedding using the same method.
3. Retrieval: The system compares the query embedding against the stored chunks and pulls back the handful that are most similar — usually the top 3 to 10.
4. Augmentation: Those retrieved chunks get inserted directly into the prompt, right alongside the user's question, before anything is sent to the LLM.
5. Generation: The LLM reads both the question and the retrieved context, then writes an answer based on what it was just handed — instead of relying only on what it memorized during training.

Why is RAG retrieval semantic, not keyword search?

The part that trips people up: retrieval isn't keyword search. It's semantic search — matching meaning, not exact words, which is why a question about "cutting costs" can retrieve a document that says "reducing expenses." The stored chunks and the query live in the same embedding space, so the system measures how close their meanings are rather than whether they share the same vocabulary.

What's the main tradeoff of RAG architecture?

The tradeoff is real: a RAG system is only as good as its retrieval step. Feed it messy chunking or a weak embedding model, and the LLM will confidently generate an answer from garbage — it has no way to know the context it was handed is wrong. That's why most of the engineering effort in a RAG system goes into the offline indexing phase, not the model itself.