Prompts that return JSON

A model that "returns JSON" often returns a paragraph, then a code fence, then JSON, then a cheerful offer to help further. Your parser wants an object and nothing else. The gap between those two outcomes is usually one or two sentences in the prompt, not a different model. This note is about making the object the entire answer.

Name the keys and the types in a sentence you can compare to the parser. "Respond with JSON. The keys are category and reason. category is one of refund, shipping, or other. reason is a string of at most 20 words. Do not add a preamble. Do not wrap the JSON in a fence." That is a specification. "Give me some JSON" is a mood. The specification is short. Paste it into the counter and you will see it is a few dozen tokens. It is the cheapest reliable part of the route. Do not bury it under a page of tone.

The preamble is the failure you will see first. The model writes "Sure, here is the JSON:" because that is how helpful answers look in ordinary chat. Your parser then fails on the letter S. Forbid the preamble in the output line, and repeat the forbid next to the format if the prompt is long. "Respond with JSON only" is stronger than "respond with JSON" because "only" is a constraint a person can see. If you use a provider setting that forces a JSON mode or a schema, turn it on and still keep the sentence. Settings and prompts drift apart when only one of them is in the repo.

Fences are the second failure. You asked for a fence yesterday because the answer was hard to read in a log, and today the parser strips nothing. Decide which consumer you have. A human log can tolerate a fence. A json.loads call cannot, unless you strip the fence yourself and you test the stripper against a fence, against no fence, and against a fence with a language tag. Stripping is application code, and it is fine. It is not a reason to be vague in the prompt. If the code expects raw JSON, say "no fence." If the code expects a fence, say "one fence, json tag, nothing before or after." Do not leave it to taste.

Show one valid object when the shape is easier to imitate than to describe. A single line is enough:

Shape: {"category":"shipping","reason":"Parcel has not scanned in four days"}

That line is an example of the format, not a few-shot lesson about the category boundary. Keep those jobs separate. The zero-shot and few-shot note explains why a pile of full tickets is a tax. A shape line is not that tax. It is twenty tokens that pin down key names and value types. Use double quotes in the example because that is what JSON requires. A model shown single quotes will sometimes emit single quotes, and a strict parser will reject them. Show the legal thing.

Enums deserve to be written out. "One of refund, shipping, or other" belongs in the prompt next to the key, not in a wiki. If you add a fourth category, the prompt is where the model hears about it. An example that uses only "refund" does not teach "other." Mention every allowed value in the instruction even if the example uses one. Then test one input per value, plus one input that should be "other" because it is tempting. The tempting case is the one that keeps you honest. "The box was crushed but I do not want anything" is not a refund if your definition says the customer must be asking for money. Put that definition in a sentence. An example can support it. An example cannot replace it.

Nested objects and arrays need a bound. "A list of issues" becomes forty issues when the document is long and the model is chatty. "Up to three issues, each with a title and a one-sentence detail" is a bound. Bounds are also how you cap output tokens, which on GPT-4-class lists cost more per token than input. A JSON answer that repeats the source document inside a field called "summary" is a prompting failure and a cost failure. Say what the field is for, and say what it must not contain. "Do not quote the source. Do not add keys that are not in the shape."

Escaping is the model's job and the parser's problem. If the customer text contains quotes and newlines, a model will sometimes break the string. You reduce the rate by keeping the customer text in the prompt as data, fenced, and by asking for short fields rather than a field that copies the email. You do not reduce it to zero. Validate. On failure, retry once with the validation error appended: "The previous answer was not valid JSON. Respond with JSON only." Log how often the retry fires. If it fires on more than a small slice, the shape is too hard or the input is too dirty for this model, and a retry loop will burn tokens to hide that.

Schema modes, where the provider accepts a JSON schema and constrains decoding, are worth using when you have them. They do not remove the need for the prompt to say what the fields mean. A schema can force category to be one of three strings and still not know that a crushed box with no request for money is "other." Meaning stays in the prompt. Syntax can move into the API. If you turn on a schema mode, delete the contradictory line that asks for a markdown table "as well." One answer, one shape.

Version the shape the way you version code. A key rename is a breaking change for every stored response. When you add a key, say whether it is required. When you remove a key, stop showing it in the example the same day. Stale examples are how old keys resurrect. The model is imitating the most answer-shaped thing in the prompt. Make sure that thing is legal under the parser you are running today, not the parser you ran in March.

A minimal prompt that holds up looks dull. Role if you need judgment. Task as a verb: classify. Context as the fenced ticket. Constraints: the enum, the ban on invented ids, the length of reason. Output: JSON only, the shape line, no preamble, no fence. Run twenty tickets. Parse all twenty in the test, not by eye. The ones that fail will teach you the next sentence. That sentence belongs in the prompt. Another adjective does not. The temperature note is the companion warning: turning the sampling temperature down will not invent the enum you forgot to write.