Chunking Documents Without Destroying Meaning

An 80-page technical manual, mechanically split every 1,000 characters. Chunk 47 ends with: the maximum allowable pressure is. Chunk 48 begins with: 12 bar, except for models prior to 2018. A user asks for the maximum allowable pressure. Vector search returns chunk 47, because it contains the words of the question. The model receives a truncated sentence, finds no value, and answers that the information is not available. You indexed the right page and lost the right answer.

This scenario is not an edge case; it is the most common failure mode of early prototypes. Chunking determines what a chunk can mean on its own, outside its document, once it has been pulled out of its context. A chunk is a self-contained unit of meaning or it is nothing: it will be retrieved alone, read alone, and cited alone. Splitting by character count means deciding that meaning stops every 1,000 characters, which is true of no document written by a human.

The Three Parameters and Their Real Trade-offs

Chunk size is a trade-off between precision and completeness. Short chunks, from 200 to 400 tokens, produce sharp vectors: the chunk is about one thing, so its embedding is unambiguous, so similarity is reliable. But it often lacks the context needed to answer. Long chunks, from 1,000 to 1,500 tokens, contain the complete answer, but their vector is an average of several topics, which makes it close to everything and nothing.