Can RAG work with SQL and structured data?

Question

Accepted Answer

Yes — RAG works with SQL and structured data, but usually not by dumping rows into a vector database. For precise, filterable data you let the model generate a SQL query against your database (text-to-SQL) or read tables directly (often called table RAG), and you use metadata filters to narrow a vector search. Vector similarity is for unstructured prose; exact queries are for structured data.

Why not just embed the whole database?

Vector search answers what's semantically similar, which is the wrong question for structured data. Total revenue in Q3 for customers in Texas isn't a similarity problem — it's a filter-and-aggregate problem, and only an exact query gets it right. Embedding rows as text tends to return records that look related but miss the precise, numeric, or aggregated answer the user actually asked for.

What are the main approaches?

Text-to-SQL: the LLM translates the question into a SQL query, runs it, and answers from the result — best for counts, sums, filters, and joins over a database. Table RAG: the model retrieves and reads relevant rows or tables directly, given the schema and column descriptions — best for question-answering over spreadsheets and tables. Metadata filtering: vector search over text, constrained by structured fields like date, author, or category — best for narrowing a document search by exact attributes. Pure vector RAG: embed and retrieve unstructured passages by meaning — best for prose like docs, articles, transcripts, and tickets.

How do I choose which one?

When the question needs exact numbers, aggregation, or filtering on fields, use text-to-SQL against the real database. When the data lives in tables but you want natural-language Q&A, use table RAG, giving the model the schema and column meanings. When it's mostly prose but tagged with structured attributes (date, region, product), use vector search plus metadata filters, a hybrid of both worlds. Many real systems combine them — a router sends factual and numeric questions to SQL and open-ended ones to vector retrieval.

The trap to avoid is forcing everything through one method. Match the retrieval style to the shape of the data, and let a router send each question to the right tool.

Approach	How it works	Best for
Text-to-SQL	The LLM translates the question into a SQL query, runs it, and answers from the result	Counts, sums, filters, and joins over a database
Table RAG	The model retrieves and reads relevant rows or tables directly, given the schema and column descriptions	Question-answering over spreadsheets and tables
Metadata filtering	Vector search over text, constrained by structured fields like date, author, or category	Narrowing a document search by exact attributes
Pure vector RAG	Embed and retrieve unstructured passages by meaning	Prose: docs, articles, transcripts, tickets