How much does a RAG system cost to run?

The cost of running a RAG system isn't one bill — it's a stack of them: embedding your documents, hosting a vector database, running retrieval on every query, and paying for LLM inference to generate each answer. On top of those visible costs sit the hidden ones — data preparation, re-indexing, and ongoing operations — which often dominate over time. What you actually pay depends on your data size and query volume, not a fixed price tag.

What are the cost drivers?

Which one usually dominates?

For most production systems, LLM inference is the biggest recurring cost, because it's paid per query and scales directly with traffic — and every retrieved chunk you stuff into the prompt adds tokens to that bill. Embedding is mostly an upfront and re-indexing cost. Vector DB hosting is a steady baseline you pay whether or not anyone's querying.

The hidden killer is data prep: cleaning, chunking, and keeping the index fresh is real engineering time that never fully ends.

What are the levers to pull?

• Retrieve less, smarter. Fewer, better chunks mean fewer prompt tokens per query — the fastest way to cut the inference bill.
• Right-size the model. A smaller model for generation, reserved for when quality actually demands the big one.
• Cache repeated queries. Semantic or exact caching skips the LLM call entirely on repeats.
• Re-index only what changed. Incremental updates beat re-embedding the whole corpus.
• Match the vector DB to scale. Don't pay for a distributed cluster when a modest index would do.

The honest summary: RAG cost is driven by volume and freshness, and the levers all come down to sending fewer tokens, calling the model less, and re-indexing only when you have to.

Estimating a specific setup? Use our free LLM API cost calculator to compare monthly spend across models.