Chunking Strategies That Actually Matter

Chunking is the most-debated RAG decision and the least-tested. The debate is usually about chunk size; the decisions that actually matter are boundaries, overlap, and structure.

Everyone argues about chunk size. Few teams test boundaries — where you cut — and that's where the retrieval quality actually lives. A 512-token chunk cut mid-sentence is worse than a 256-token chunk cut at a boundary.

Strip it down: chunking exists to surface the right unit of context to the model. The unit that matters is the one that's self-contained enough to be useful and small enough to be precise. Size is a means; the self-contained unit is the end.

• Self-contained: a chunk that makes sense without its neighbors.
• Precise: a chunk small enough to point at the right answer.
• Structured: a chunk that respects document structure (headings, sections).

Issue-tree the chunking decisions that actually move retrieval quality:

• **Boundaries.** Cut at structural boundaries (sentence, paragraph, heading), not token count.
• **Overlap.** A small overlap so a concept split across chunks isn't lost.
• **Structure.** Preserve headings and section context with the chunk.

Chunk size is the last of these, not the first — and it should be chosen after boundaries, not before.

Chunking changes must be evaled, not vibes-checked. Run the same eval set across chunking strategies and compare retrieval recall and precision. The strategy that wins on the eval wins, regardless of what "feels right."

• Fixed token chunks cut mid-sentence.
• No overlap, so split concepts are lost.
• No structure preserved, so a chunk loses its section context.
• Optimizing chunk size without first fixing boundaries.

Chunking quality lives in boundaries, overlap, and structure — not size. Choose the self-contained unit first, eval every strategy, and treat chunk size as the last decision, not the first.