What is the best chunk size for RAG?

There's no single best chunk size, but a solid default is roughly 256 to 512 tokens per chunk with about 10 to 20 percent overlap — then tune from there. Smaller chunks (128 to 256 tokens) give precise, focused retrieval for fact lookups; larger chunks (512 to 1024) preserve more context for questions that need reasoning across a passage. The right number depends on your documents and the kind of questions you answer.

Why does chunk size matter so much?

Chunking is a trade-off between precision and context. Tiny chunks match a query tightly and waste few tokens, but they can slice a single idea in half, so the answer ends up split across two chunks and neither retrieves cleanly. Big chunks keep ideas whole but dilute the embedding — one vector now has to represent several topics at once, so it matches queries less sharply and drags in irrelevant text.

You're really tuning where that balance should sit for your content.

What levers actually help?

• Overlap — repeat 10 to 20 percent of tokens between neighboring chunks so a sentence that straddles a boundary isn't lost.
• Semantic vs fixed chunking — fixed-size splitting is simple and predictable; semantic chunking splits on natural boundaries (headings, paragraphs, topic shifts) and usually retrieves better because each chunk is self-contained.
• Respect structure — split on document structure like headings, list items, and table rows rather than blindly every N tokens.
• Match the models — keep chunks inside your embedding model's context window, and remember the LLM has to fit all the retrieved chunks in its prompt too.

How do I pick for my case?

Let the question type decide:

• Fact lookup, FAQ, short answers → smaller chunks (128 to 256 tokens), because you want one precise passage.
• Reasoning, summaries, how-does-X-relate-to-Y → larger chunks (512+ tokens) so each carries enough context to reason over.
• Mixed workloads → start around 400 tokens with overlap, build a small eval set of real questions, and adjust from what you see.

Don't overthink it up front. Pick a sensible default, measure retrieval on real queries, and change one variable at a time so you can tell what actually helped.