devtools.codes

How to count tokens in text

Your tool input is processed locally in your browser and is not intentionally uploaded to our servers. Advertising and analytics providers may still process normal page, device, cookie and network information.

A token is not a word and not a character. Tokenisers split text into subword units learned from a training corpus, so common words usually become a single token while rare words, names and code identifiers break into several. That is why token count and word count drift apart, and why the gap widens for technical text.

The rough guide most people use — about four characters per token for English prose — is reasonable for planning and unreliable for anything else. It underestimates code, which is dense with punctuation and unusual identifiers. It underestimates languages that do not use Latin script, sometimes by a factor of two or more. And it says nothing about the system prompt, tool definitions and message scaffolding that a provider counts alongside your text.

Counts also differ between providers. Each model family ships its own tokeniser with its own vocabulary, so identical text produces different counts on different models. A prompt that fits comfortably in one context window can overflow another of nominally the same size. If you are close to a limit, count against the model you are actually calling.

The practical reasons to count are budgeting, context-window planning and debugging. For budgeting, remember that input and output are priced separately and output is usually several times more expensive. For context planning, leave real headroom rather than filling the window to its stated capacity. For debugging, a sudden jump in token count is often the first visible sign that a retrieval step is returning more than you expected.

Counting in the browser has one further advantage worth stating plainly: the text you are measuring is frequently a production prompt or a customer record. Pasting that into a remote service to get a number is a poor trade.

More on how to count tokens in text

How many tokens is 1,000 words?

For ordinary English prose, roughly 1,300 to 1,400 tokens, because the common four-characters-per-token guide works out at about three-quarters of a word per token. Treat that as a planning figure and nothing more. Technical writing, code samples, unusual names and any non-Latin script all push the count higher, sometimes by half again.

Do spaces and punctuation count as tokens?

They are counted, though rarely as tokens of their own. Most modern tokenisers attach a leading space to the word that follows, so "the cat" is usually two tokens rather than three. Punctuation is frequently merged into an adjacent piece as well. This is one reason character-based estimates drift: the mapping is learned, not mechanical.

Does the system prompt count towards my token usage?

Yes, and on every single request. The system prompt, any tool or function definitions, and the full conversation history so far are all sent with each call and all billed as input. For a short user message with a long system prompt, the system prompt can easily be the larger part of the bill.

Why does the same text give different counts on different models?

Each model family ships its own tokeniser with its own learned vocabulary, so identical text splits differently. A word that is one token for one model may be three for another. This matters most near a context limit: a prompt that fits one model comfortably can overflow another with the same advertised window size.

The tool behind this, and related guides