devtools.codes

How to count tokens in text

Not built yet · Stage 4 — blocked

The calculator this page depends on is not built yet. It needs a pricing data source (OQ-04) and confirmation that the tokeniser licences permit commercial use (OQ-05).

Your tool input is processed locally in your browser and is not intentionally uploaded to our servers. Advertising and analytics providers may still process normal page, device, cookie and network information.

A token is not a word and not a character. Tokenisers split text into subword units learned from a training corpus, so common words usually become a single token while rare words, names and code identifiers break into several. That is why token count and word count drift apart, and why the gap widens for technical text.

The rough guide most people use — about four characters per token for English prose — is reasonable for planning and unreliable for anything else. It underestimates code, which is dense with punctuation and unusual identifiers. It underestimates languages that do not use Latin script, sometimes by a factor of two or more. And it says nothing about the system prompt, tool definitions and message scaffolding that a provider counts alongside your text.

Counts also differ between providers. Each model family ships its own tokeniser with its own vocabulary, so identical text produces different counts on different models. A prompt that fits comfortably in one context window can overflow another of nominally the same size. If you are close to a limit, count against the model you are actually calling.

The practical reasons to count are budgeting, context-window planning and debugging. For budgeting, remember that input and output are priced separately and output is usually several times more expensive. For context planning, leave real headroom rather than filling the window to its stated capacity. For debugging, a sudden jump in token count is often the first visible sign that a retrieval step is returning more than you expected.

Counting in the browser has one further advantage worth stating plainly: the text you are measuring is frequently a production prompt or a customer record. Pasting that into a remote service to get a number is a poor trade.

Related