cl100k_base is the tokenizer vocabulary behind GPT-4-class models. The name is literal enough: a vocabulary on the order of a hundred thousand chunks, built with byte-pair encoding. When a usage page says you sent 1,240 tokens, it means 1,240 of those chunks, not 1,240 words and not 1,240 characters divided by four. The rule of thumb is a decent average across a novel. It is a poor description of a prompt, because prompts are full of the cases the average hides.
The first cut is a regular expression, the pretokenizer, before any merge is applied. It peels off contractions such as "'t", "'ll", "'s", and "'re". It takes a run of letters, optionally with one character glued to the front, which is why " world" with a leading space is one piece and why "/path" is one piece. It takes digits in groups of one to three, so "1000" is not one chunk at this stage. It takes symbol runs and whitespace, and a newline often sticks to the punctuation in front of it. Then byte-pair encoding either accepts each piece as a single token or splits it again. For ordinary English the first cut is already the token. "The quick brown fox jumps over the lazy dog." is ten pieces and ten tokens. "Hello, world!" is four: "Hello", ",", " world", "!".
The leading space is the fact that breaks homemade counters. In the middle of a sentence the space is usually part of the word's token. At the start of a line there is no space, and many words that are one token with a space are several tokens without it. " summarize" with a leading space is one cl100k token. "summarize" at the start of a line is three, along the lines of "sum", "mar", "ize". "Rewrite" at the start of a line is two. " rewrite" with a space is one. "Analyze" at the start is two. A prompt that begins every line with an imperative verb is systematically more expensive than the same words in a paragraph, and a counter that only looks at character length will miss the difference in both directions.
Capitalization is a different token, not a flag on the same token. "hello" and "Hello" can each be one token and still be different ids. That does not change your bill by itself, since each is one. It changes the bill when the capitalized form was never merged and the lowercase form was. The practical habit is to count the string you will send, in the case you will send, rather than a normalized copy. The estimate on this site keeps a list of common words whose line-start form and whose spaced form disagree, because a length rule cannot see that "Summarize" is three and " summarize" is one.
Rare and long words fall through to several tokens even with a space. "tokenization" is two tokens with or without the leading space: roughly "token" and "ization". "standardization" splits the same way. "configuration" and "information" do not. There is no visual rule that tells you which is which, short of the vocabulary. Very long coinages split into many pieces. "antidisestablishmentarianism" is six tokens. The playful "supercalifragilisticexpialidocious" is eleven. If your domain is full of identifiers like these, sample them. A product name you invented will not be one token just because it is one word in your head.
Digits and punctuation are smaller than people budget and occasionally larger. "12" is one token. "123" is one. "1234" is two, because the pretokenizer stops at three digits. Prices like "$12.50" break into the dollar sign, the dollars, the dot, and the cents. A URL breaks into the scheme, the slashes, the host, the path segments, and the query. "https://example.com/path/to/resource?q=1&x=2" is on the order of fourteen tokens, not one. If you paste twenty URLs into a prompt so the model can "look at them," you are pasting a few hundred tokens of addresses and no page content. The model cannot fetch them unless you have a tool for that. You paid for the characters of the link.
Whitespace is usually cheap and occasionally merged. Two spaces before a word often become one token for the extra space plus one token for the word-with-space. A blank line is not free, but it is not a paragraph either. Indentation in code is tokens. A Python function with a docstring and a four-space indent costs more than the same logic described in a sentence, and it also costs whatever the model spends quoting the code back. Paste code when the task is about that code. Do not paste a file to set the mood.
Other scripts are a different map. cl100k was merged on a corpus that is heavier in English than in Hindi. A Devanagari sentence can land near one token per Unicode code point, which is much less efficient than English words. We checked "कृपया इस दस्तावेज़ का सार लिखें और तीन बिंदु दें।" at 52 real cl100k tokens. A short Japanese greeting can be a single token if that sequence was merged, and the next kanji can break the run. Emoji are often two or three tokens for one character. If your users write in a script you have not sampled, sample twenty of their messages in the real tokenizer before you multiply by daily volume. The English 5 percent band this site quotes does not automatically apply.
What the count does not include is the rest of the API request. Tool definitions, image inputs, and the tokens the model spends on the way to an answer are outside the string you pasted in a box. Output is billed at a higher rate on every current GPT-4-class list we have checked. A perfect input count can still describe a minority of the bill. Use cl100k to compare two versions of the instruction. Use the usage object on the response to see what you were charged. The cost note is that multiplication, with a worked day.
If you remember three mechanics, remember these. The space in front of a word is usually part of the token. A word at the start of a line may split even when the same word mid-sentence does not. Digits group by threes, and a URL is many tokens. Everything else is worth checking once on your own text, then leaving alone. Chasing a one-token difference in a conjunction is how people avoid the 800-token paragraph they paste twice.
The estimate on the home page follows the pretokenizer and then corrects the splits that show up in ordinary English: line-start verbs, a few always-split words, and very long rare words. It matched the real encoder exactly on the fox sentence, on "Hello, world!", on a small JSON object, and on both of the long coinages above. It was about 3.6 percent low on a URL-heavy sentence and about 2 percent low on a markdown explainer of roughly 1,500 tokens. That is a planning tool. It is not the usage page, and it should not be described as one. The filler note is what to do after you can see the number move.
Questions
Why is my word count so different from the token count?
Because tokens are merged chunks, not words. English is often a bit more than one token per word once you add punctuation and line-start splits. Code, URLs, and non-English text can be much further from the word count. Count the string. Do not multiply words by 1.3 and call it a day if the string is not prose.
Does a new GPT model use the same tokenizer?
GPT-4-class chat models have stayed on cl100k_base for a long time, but a new model can ship a new tokenizer. When a provider says the encoding changed, recount. Do not keep a spreadsheet of cl100k figures for a model that no longer uses it.
Are spaces at the end of the prompt free?
No. Trailing whitespace is usually a token or part of one. It rarely matters. A trailing copy of your entire policy, which is also "whitespace" in the sense that nobody reads it twice, matters a great deal.