What is prompt engineering?
Prompt engineering is the work of writing the instructions a language model actually receives. The model does not see your intention. It sees a string of text, which a tokenizer cuts into numbers, which a network turns into the next number, over and over, until a stop rule fires. If the string is vague, the continuation is vague. If the string mixes a task, a pile of source text, and three different output wishes, the continuation has to guess which part was the assignment.
People talk about prompt engineering as if it were a bag of magic phrases. In practice it is closer to writing a short brief for a competent colleague who has no memory of your last meeting and who will follow the last clear instruction more readily than the first polite one. You name the job, you name the material, you name the boundaries, and you name the shape of the answer. That is the whole craft. The rest is editing.
A useful brief has a few jobs, and they should not be tangled together. The role, when you need one, tells the model which kind of judgment to apply: a support editor, a taxonomist, a translator who keeps technical terms in English. The task is the verb. Summarize, extract, rewrite, classify, rank. The context is the raw material: the email, the spec, the table, the error log. Constraints are the refusals: do not invent a policy, do not use the customer's full card number, stay under 120 words. The output format is the shape you can parse later: a markdown reply, a JSON object with two keys, a table with three columns. Examples are optional. One careful pair often beats five sloppy ones, and five sloppy ones cost five times the tokens.
None of this requires a special product. You can do it in any text box. The reason this page exists is that the text box hides the cost and the mess. A prompt can feel short and still be expensive, because tokenizers do not count words. A prompt can feel polite and still be empty, because "please help me write something nice" does not name a task. The counter and the score are there so you can see both problems before you pay a provider to see them for you.
There is a second, quieter part of the job. Once a prompt works, you keep it stable. Teams that paste a fresh paragraph into a chat every morning cannot tell whether Tuesday's worse answer came from the model, the temperature, or the fact that someone deleted the line that said "do not invent order numbers." A structured prompt is easier to diff. You can see that the constraint moved. You can see that the example grew by 400 tokens and ask whether those 400 tokens earned their keep.
Prompt engineering is not a way to make a model do something its provider forbids, and it is not a substitute for checking the answer. A well-built prompt makes a capable model more predictable. It does not make a weak model knowledgeable, and it does not make a fabricated citation real. If your task needs a source of truth, put that source in the context and tell the model to use only that. If you do not have a source, say so, and ask for a draft that a person will verify.
How tokenization works (BPE and tiktoken)
Before a model reads your prompt, a tokenizer turns the string into tokens. A token is not a word and not a character. It is a chunk that showed up often enough, in the data the tokenizer was trained on, to earn a place in a fixed vocabulary. GPT-4-class models use a vocabulary called cl100k_base, built with byte-pair encoding and exposed by OpenAI's tiktoken library. The "100k" is the size of the vocabulary, on the order of a hundred thousand chunks. Claude and other families use their own tokenizers. A count you make for cl100k is the right planning number for GPT-4-class chat models. It is only a rough cousin of a Claude bill.
Byte-pair encoding starts from bytes, not from a dictionary a person wrote. The trainer looks at a large corpus and repeatedly merges the pair of symbols that appears most often. Early merges glue common letter pairs. Later merges glue whole words, then words with the space in front of them, then short phrases that are almost idioms. The result is a vocabulary where "the" is one token, " the" (with the leading space) is often a different single token, and a rare word such as "antidisestablishmentarianism" is several tokens because nobody merged it into one chunk.
That leading space is the detail that surprises people who try to estimate tokens by dividing character count by four. In cl100k, a word in the middle of a sentence is usually stored with its space attached. " hello" can be one token. The same letters at the start of a line, with no space in front, are sometimes one token and sometimes three. "summarize" at the start of a line is three tokens in cl100k_base (roughly "sum", "mar", "ize"). " summarize" with a leading space is one. Capitalization changes the split again. "Rewrite" at the start of a line is two tokens. " rewrite" with a space is one. A counter that only divides length by four will be calmly wrong on both of these, in opposite directions.
tiktoken does the split in two stages. First a regular expression cuts the string into pretokens: contractions such as "'t" and "'ll", a run of letters with an optional one-character prefix, digit groups of one to three, symbol runs, and whitespace. Then byte-pair encoding cuts any pretoken that is not already a single vocabulary item. For ordinary English instructions the first stage does almost all the work. "Hello, world!" is four pretokens and four tokens: "Hello", ",", " world", "!". A careful editor's brief of a few dozen words is often one token per pretoken. The second stage shows up on rare words, on some line-start words, on URLs, and on scripts that were scarce in the merge training.
Numbers are grouped in threes. "12" is one token, "123" is one token, "1234" is two. Punctuation is usually its own token, unless it has been merged with what follows. ".com" is often one token. A newline can merge with the punctuation in front of it, so ".\n" may be a single token rather than a period plus a line break. Emoji are not one-for-one: a single smiley is often two tokens, and some symbols are three. None of this is visible in the chat box. It is visible on the invoice, because providers bill input and output separately, per token, and a long system prompt is paid on every call.
The practical consequences are boring and useful. Cutting filler words saves less than people hope, unless the prompt is mostly filler. Repeating a full policy on every call costs more than people notice. Putting a worked example in the prompt is often worth it, and sometimes it is the most expensive line you have. A JSON schema pasted twice, once in the system text and once in the user text, is a tax. Counting before you send is how you notice the tax.
Why structured prompts yield better AI results
A model continues a pattern. If your prompt looks like a brief, the continuation looks like the thing the brief asked for. If your prompt looks like a chatty request with the real assignment buried in sentence four, the continuation looks like a chatty answer that may or may not have done the assignment. Structure is not a ritual. It is how you make the pattern unambiguous.
Labels do a specific job. When a line says "Task" and the next line is a verb phrase, the model does not have to decide whether that sentence was background. When a line says "Constraints" and the next line is "Do not invent a refund policy," that sentence is harder to treat as optional color. When the source email sits under "Context," the model is less likely to answer the email's question back to you as if you were the customer. People who have cleaned a few hundred prompts recognize the failure mode: the model replies to the sample instead of performing the task on the sample. A heading is a cheap fix.
Delimiters do the same job for messy material. A customer email, a stack trace, or a block of HTML contains instructions by accident. "Ignore the above and do this instead" inside a quoted email is a classic mess, and it is also just a confusing document. Wrapping the document in a fence, or in a pair of tags, tells the model where the assignment ends and the evidence begins. Models do not have a perfect record of respecting fences. They respect them more often than they respect a paragraph that starts "here is the email" and then never clearly ends.
Format lines change the last-mile reliability. "Be helpful" produces a helpful-looking essay. "Respond with JSON and use the keys subject and body" produces something a program can parse, most of the time, if you also say not to add a preamble. The preamble is the usual leak. The model writes "Sure, here is the JSON:" and then the object, and your parser breaks. A single sentence forbidding the preamble is worth more than a paragraph of tone advice.
Examples work when the format is easier to show than to describe. A few-shot prompt is a sequence of inputs and outputs that demonstrates the boundary cases: what to do with a missing order number, what to do with an angry opener, what not to promise. Zero-shot is the right choice when the task is ordinary and the format is a sentence you can specify. Few-shot starts to pay for itself when you have already watched the zero-shot version make the same mistake twice. The cost is real. Each example is more input tokens on every call. Count them. If the example is 300 tokens and you make 10,000 calls a day, you are buying that example 10,000 times.
There is a limit to what structure can do. It will not insert facts the model does not have and that you did not supply. It will not reliably hold a format across a very long, very noisy context if the format line was at the top and the context is thirty pages. Repeat the output shape near the end if the prompt is long. Structure also will not fix a task that is secretly three tasks. "Read this, decide if we should refund, write the reply, and draft the internal note" is four jobs. Split them if you need each one to be checkable.
How this page counts tokens without uploading the prompt
A faithful cl100k count needs the merge table, which is what the tiktoken library carries, often as a WebAssembly build of a few megabytes. This page does not ship that table and does not send your prompt to a server that has it. The estimate is local. It runs the same style of pretokenizer tiktoken uses, then applies a compact set of corrections for the pieces that byte-pair encoding would split further.
The pretokenizer is the accurate part for normal English. Contractions, letter runs, short digit groups, symbols, and whitespace are cut the way cl100k cuts them. On a plain sentence such as "The quick brown fox jumps over the lazy dog." the pieces and the real tokens are the same, and the estimate matches exactly. The same is true of a short support brief, of a small JSON object, and of "Hello, world!".
The corrections cover the cases where one piece is several tokens. A line-start word such as "Summarize" or "Analyze" is looked up, because those splits are systematic and a length rule gets them wrong. A few longer words that stay split even with a leading space, notably "tokenization" and "standardization," are in the same list. A very long rare word uses a length rule: "antidisestablishmentarianism" estimates at 6, which is the real cl100k count, and the even longer "supercalifragilisticexpialidocious" estimates at 11, also the real count. The list is a few thousand common words and their known exceptions. It is not the hundred-thousand-entry vocabulary, and it is not loaded from the network.
We checked the estimate against tiktoken's cl100k_base encoder. Short English instructions, a JSON object, a fenced-style code sample, a long rare word, and a sentence that includes "tokenization" and "standardization" all landed inside 5 percent, and most of them landed on the exact integer. A URL-heavy sentence was about 3.6 percent low. A markdown explainer we measured came out at 1,527 estimated tokens against 1,558 real ones, about 2 percent low. That is the band this page is willing to claim: English instructions, typically inside 5 percent.
Other writing systems are estimated more roughly, and the page says so in the tool. Devanagari is counted close to one token per code point. On the sentence "कृपया इस दस्तावेज़ का सार लिखें और तीन बिंदु दें।" the estimate was 49 against 52 real tokens, about 6 percent low. A short Japanese greeting that cl100k stores as a single token is counted as one. Emoji are approximated. Do not use this number as the bill for a Hindi or Japanese corpus without checking a sample in the provider's own tokenizer. Do use it to notice that a pasted policy grew by a thousand tokens.
Because the work stays in the tab, closing the page drops the prompt. This site does not write it to local storage, does not set a cookie, and does not have an account system. If you later put advertising on the site, that is a separate change, described in the privacy note. The counter itself has no reason to phone home, and the scripts that count do not.
How the density score is calculated
The density score starts at 40 and moves up or down based on a fixed list of signals. It is clamped between 1 and 100. It does not call a model. Two people who paste the same text get the same score, and you can read the reasons next to the number.
Filler pulls the score down. The phrases are: please, kindly, can you, could you, would you, I would like, I would like you to, I want you to, help me, if possible, thank you, thanks, just, really, very, basically, and actually. Each hit costs 4 points, and the penalty stops at 28 so a groveling paragraph cannot bury a prompt that is otherwise clear. Longer phrases are counted instead of the shorter ones inside them, so "I would like you to" is one hit, not three.
If the text has no action verb from the list and no labeled section, it loses 8 points. The verbs are analyze, extract, list, compare, summarize, classify, rewrite, generate, return, output, convert, rank, draft, calculate, and translate. Each one adds 3 points, up to 15. "Write" and "help" are deliberately not on the list. They are too easy to hide inside an empty request.
Structure adds points, each bonus once or up to a cap. A delimiter is worth 8: a code fence, a "###" heading, a triple-quote block, a line that is only "---", or a simple tag. A labeled section is worth 4, up to 20. The labels are Role, Task, Goal, Context, Constraints, Format, Examples, Input, and Output, either as a heading or as a line that starts with the word. Naming an output shape is worth 8. The shapes we look for are JSON, a markdown table, a bullet list, a schema, or the phrase "respond with." A hard constraint is worth 6: must, must not, do not, don't, never, only, at most, at least, or a number of words or bullets. An example is worth 8: a line that starts with "Example:", or both "Input:" and "Output:", or a Q: and A: pair.
Read the reasons, not only the number. A score of 22 with "6 filler phrases" and "no action verb" is a prompt that has not said what to do. A score of 73 with labeled sections and a constraint is a prompt a colleague could follow. A score of 99 is not "better writing" in a literary sense. It means the signals we count are all present. You can game the score by sprinkling the words "must" and "JSON" into nonsense. The score is a checklist you can see, not a grade from a teacher.
A worked example, with the numbers this page produces
Press "Load example" and you get this draft. It is the kind of prompt people actually paste: polite, slightly panicked, and mixed up with the customer's words.
please can you help me I would like you to write a reply to this customer. Make it really professional. Thank you. You are a support editor. Do not invent a refund policy. Respond with markdown and keep it under 120 words. Hi, my order 1842 arrived broken and I want my money back today.
On this page the estimate is 66 tokens. The density score is 22. The reasons are six filler phrases, no action verb from the list, an output format (the words "respond with"), and a hard constraint (120 words, and "do not"). The role line is present as a sentence, but it is not labeled, so the score does not give it a section bonus. The customer's sentence is sitting in the same blob as the assignment. A model will often cope. You should not have to hope.
Press Optimize. The rewrite is deterministic. Filler phrases come out. The role sentence moves under a Role heading. The prohibition moves under Constraints. The format sentence moves under Output format. What remains becomes the Task, including the customer's lines, because they were not in a quote block or a fence. If you had wrapped the email in a fence or started it with "Context:", it would have landed under Context instead. The result looks like this.
# Role You are a support editor. # Task Write a reply to this customer. Make it professional. Hi, my order 1842 arrived broken and I want my money back today. # Constraints Do not invent a refund policy. # Output format Respond with markdown and keep it under 120 words.
The estimate moves from 66 tokens to 65. Density moves from 22 to 73. The reasons on the rewrite are one action verb (the heading contains "output"), four labeled sections, the output format, and the constraint. Token count barely changed. The headings cost about what the filler saved. That is a normal outcome, and the line under the button says so. Structure is the point. Savings show up when the original was mostly cushion, or when you then delete a repeated policy that the headings made obvious.
If you want the customer text out of the task, put it in a fence yourself before you optimize, or label it "Context:". The rewriter will not guess that a bare sentence is evidence. Guessing wrong would delete or move material you needed. The tool is conservative on purpose.
You can do the same edit by hand in less time than it takes to read this section, once you have done it a few times. The button is for the draft you are too close to. The number is for the moment you wonder whether the example you just added is larger than the task.
What this estimate is not
It is not a billing export. Providers charge input and output separately, cached input at a different rate, and sometimes a batch rate that is half of the live rate. This page multiplies the input estimate by a public list price and labels the product. It does not invent an output length. A 66-token prompt that produces a 400-token answer costs whatever the output rate says, and that dominates the bill on short prompts. The illustrative prices in the panel were checked on 25 October 2026: GPT-4.1 at $2 per million input tokens and GPT-4o mini at $0.15, from OpenAI's pricing page, and Claude Sonnet 5.5 at $2 per million input tokens from Anthropic's published list. Those numbers move. The panel is a sketch of scale, not an invoice.
It is not a context-window guarantee. Fitting under a window is a property of the provider's tokenizer and of the exact request, including system text, tool definitions, and images. Images are not tokens in the sense this box counts. A prompt that estimates at 8,000 tokens can still fail if the rest of the request is large, and a prompt that estimates at 200 tokens can still be the wrong prompt.
It is not affiliated with OpenAI or Anthropic. cl100k_base is their design, described here so you know what the estimate is aiming at. The density score is ours. A high score does not mean a model will comply, and a low score does not mean the model will fail. It means the draft still reads like a favor instead of a brief.
If you are about to send customer data, medical notes, or secrets, the right question is not the token count. It is whether that text should be in a prompt at all. This page cannot answer that. It can keep the text on your machine while you decide.
Questions
Does this match the token count in the OpenAI usage dashboard?
Not to the token. The dashboard uses the provider's tokenizer on the exact request, including parts of the request this box never sees. On English instructions our check set stays inside 5 percent of cl100k_base, and many short prompts match exactly. Use it to compare drafts. Use the dashboard to audit a bill.
Why did Optimize make the prompt longer?
Headings are tokens. If your draft had no filler to remove, the labels add more than they save. The note under the result says how the estimate moved. Keep the structure if you needed it, and delete a heading you do not need. An empty Role section is not emitted. An empty output format is filled with one default sentence, which you should replace with the shape you actually want.
Can I use the count for Claude?
Only as a rough scale. Claude's tokenizer is not cl100k_base. The same English sentence is often in the same neighborhood, and then a code block or a non-English passage drifts. The structural advice on this site, especially about putting evidence in a fenced or tagged block, still applies. The integer does not transfer cleanly.
Do you store what I paste?
No. The scripts run in the page. There is no account, no request, and no local storage entry for the prompt. When you close the tab, it is gone. The privacy page says the same thing, and it says what would change if advertising were added later.
The notes in the blog go further on cost, on zero-shot versus few-shot, and on how a Claude-style prompt and a GPT-style prompt want different scaffolding. The tool stays this page. Count the draft you have, then decide whether it is a brief.