What is chunking in RAG?

Chunking is the step where you cut a source document into smaller pieces before turning it into embeddings and storing it in your vector index, so retrieval can hand back the one paragraph that answers a question instead of the entire file.

Here's why it matters: an embedding model squeezes a piece of text into a single vector. Feed it a twenty-page document and you get one vector that's an average of everything in it — vaguely about a lot of things, specifically about nothing. Chunk that same document into paragraph-sized pieces and each chunk gets its own vector that actually represents one idea, so a question about a return policy retrieves the return-policy paragraph, not the whole employee handbook.

Common chunking strategies

• Fixed-size: split every N tokens or characters, usually with some overlap between chunks so a sentence straddling a boundary doesn't lose its context.
• Sentence or paragraph-based: split on natural breaks so you never cut a thought in half.
• Semantic: split wherever the meaning shifts, using an embedding model to detect where one topic ends and the next begins.
• Recursive or structure-aware: split along headers, bullet lists, or code blocks first, then fall back to fixed-size chunking for anything still too long.

The tradeoff cuts both ways. Chunks too big and you're back to vague, averaged-out vectors that drag in reams of irrelevant text along with the answer. Chunks too small and you strip away the surrounding context a sentence needs to make sense, so the model gets a fragment without the full picture.

There's no universal right size — it depends on your content and how much context your model can hold. Treat chunk size and overlap as settings you tune against your own documents, not a number you copy from someone else's blog post.