Count tokens before a batch job

A batch job is one prompt wearing a thousand inputs. Whatever you got wrong in the template, you get wrong at volume, and whatever you left in the template, you buy at volume. Counting tokens before the launch is not a nicety. It is the difference between a bill you predicted and a bill you explain later. The arithmetic is ordinary. The discipline is doing it on the template you will actually send, with real rows, not on a sketch.

Take the template and mark the slot where the row goes. Count the template with the slot empty. That number is the overhead, paid on every row. Then count ten real rows, including the longest one you can find, not the tidy one from the spec. Records have a long tail: a note field that is empty in the demo and 2,000 characters in row 14,003. If you only measure the demo, you will underestimate the tail and you may also overflow a context window on exactly the rows that were messy enough to need the model. The cl100k estimate is the right tool for the overhead if the model is GPT-4-class and the template is English. Check a sample of rows that are not English in the provider's tokenizer. The overhead estimate can be excellent while the tail is a different script and a different ratio.

Write the multiplication where you can see it. Overhead 800 tokens, typical row 150, output cap 120, 40,000 rows. Input is about 950 × 40,000 = 38 million tokens. At GPT-4.1's $2 per million list price that is $76 of input, before cache and before batch discount. Output, if you hit the cap on every row, is 120 × 40,000 = 4.8 million tokens. At $8 per million that is $38.4. You will not hit the cap on every row. You also will not hit zero. Sample fifty rows, average the actual output tokens from a pilot, and use that average instead of the cap. The cap is the worst case you set. The pilot is the expectation. Report both.

OpenAI's published Batch API rate has been about half the live rate for jobs that can wait, as described on their pricing page when we checked in October 2026. Half of $114 is still worth knowing before you click, and the discount is not a reason to skip the count. A template that is twice as long as it needs to be is twice as long at the discount too. Confirm the discount applies to the model you are calling, and confirm your client is actually using the batch endpoint. A loop of live calls is not a batch, even if you run it overnight.

Pilot fifty rows before the 40,000. Parse every output. The failures in fifty are the failures you would have paid to generate 40,000 times. If the failure is a preamble before JSON, fix the template, do not start the batch and write a cleaner. If the failure is a row with no text, decide what the job should emit for empty input and put that in the template as a rule, so you do not spend a full generation discovering the emptiness. If the failure is the long-tail row that blew the window, truncate the field on purpose, with a marker that it was truncated, rather than discovering the error at row 14,003 after the easy rows have already been billed.

Idempotency and retries change the token total. A job that retries the whole batch on any error can double the bill. A job that retries one row is fine. Know which one you built. Count a retry as another full input, because that is what it is. The same is true of a two-pass design: a cheap model filters, an expensive model handles the rest. Add the filter tokens to the total. People compare the expensive model's price and forget that the filter saw every row. The cost note is the same arithmetic for online traffic. Batch just makes the multiplier impossible to wave away.

Output caps are part of the count. Set one. An uncapped batch classification will sometimes write a paragraph where you asked for a label, and you will pay for the paragraph on whatever fraction of rows the model feels expansive. "One label from this list, nothing else" plus a max-token setting is a belt and braces. The setting is the hard stop. The sentence is what keeps the answer inside the stop so you do not get truncated JSON. Truncated JSON is a failed row you still paid for. Look at the last token of a capped response in the pilot. If you are hitting the cap, the cap is too low or the prompt is asking for too much. Fix that before the full run.

Store the count next to the job. Template version, overhead tokens, pilot average input, pilot average output, row count, model, price assumption, estimated total, actual total when the job ends. The actual total is the provider's usage sum, not your estimate. If they disagree by more than you expected, the template you counted is not the template you sent. A wrapper added a timestamp, a tool list, or a system prompt you forgot. Finding that discrepancy once is worth more than refining the estimate. The estimate's job is to catch a 10× mistake before launch. The usage sum's job is to be true.

Do not launch a batch to "see what the prompt wants to do." See what it wants to do on fifty rows you can read. A batch is a commitment that the template is done. Done means: the slot is marked, the fence is around the row, the output shape is one shape, the empty-row behavior is defined, the cap is set, and the multiplication fits the budget you have. The shortening note will not save a batch you were not willing to sample. Counting will.

If the multiplication does not fit, cut the overhead first. A shorter template is a saving multiplied by every row, which is the only kind of saving that matters here. Cut a duplicated instruction, not the constraint that defines a bad row. Then run the fifty again. Launch when the pilot is boring. Boring means every row parsed, the tail fit in the window, and the total looked like the multiplication. That is the standard. A batch that is exciting is a batch that is wrong.