devtools.codes

Choosing the best chunk size for RAG

Your tool input is processed locally in your browser and is not intentionally uploaded to our servers. Advertising and analytics providers may still process normal page, device, cookie and network information.

There is no universally correct chunk size, and any article offering one number is guessing on your behalf. What there is, is a trade-off you can reason about and then test against your own material.

Small chunks retrieve precisely. A two-hundred-character chunk matching a query is almost certainly about that query. But it often lacks the context that makes it useful: a sentence describing a configuration option, retrieved without the paragraph explaining what the option is for, may be accurate and still unhelpful to the model reading it.

Large chunks carry context and retrieve bluntly. A two-thousand-character chunk contains the surrounding explanation, but matches a wider range of queries less precisely, and fills the context window with material that is mostly irrelevant to the specific question.

The practical range for prose is usually somewhere between three hundred and eight hundred characters, but the structure of your material matters more than the number. Documentation with short, self-contained sections chunks well at natural boundaries. Continuous narrative prose does not, and needs overlap to avoid cutting mid-argument. Reference material with dense tables often needs a different approach entirely.

Choose a strategy before choosing a size. A recursive strategy that respects paragraph boundaries produces chunks of varying length that mostly end where a thought ends. A fixed-size strategy produces predictable lengths and cuts wherever the counter lands, sometimes mid-word. For most prose the first is better, and the variance in chunk length is a feature rather than a problem.

Then test it. Run your actual corpus through two or three settings and read the chunks. Boundaries that land mid-sentence, chunks that consist mostly of a heading, and chunks that duplicate their neighbour are all visible immediately and invisible in any aggregate metric.

More on choosing the best chunk size for RAG

Should chunk size be measured in characters or tokens?

Tokens are the meaningful unit, since the embedding model has a token limit and your context budget is counted in tokens. Characters are easier to reason about and adequate for prose, where the ratio is fairly stable. For code or mixed scripts the ratio moves enough that a character limit can overshoot a token limit without warning.

Does chunk size need to match my embedding model limit?

It must not exceed it, or the tail of each chunk is silently truncated before embedding — the text is indexed as though the ending were not there. Staying well under the limit is usually right anyway: embedding models tend to represent a focused passage better than one filled to capacity.

Should headings be repeated in every chunk?

For structured documentation, prepending the section heading to each chunk is often the single cheapest improvement available. It costs a few tokens and gives an otherwise anonymous passage the context that makes it retrievable, which matters when several sections use similar wording.

How do I know when the size is wrong?

Read the chunks rather than a metric. Boundaries landing mid-sentence, chunks that are mostly a heading with a line under it, and neighbouring chunks that read as the same content are all obvious on sight and invisible in any aggregate score. If a chunk would not answer its own query on its own, it is too small.

The tool behind this, and related guides