Structured output and constrained decoding

• Logit — the raw score the model assigns each vocabulary token for the next position; softmax turns logits into probabilities.
• Vocabulary / token — the model’s fixed list of text chunks; a token can be a whole word, a fragment, or a few punctuation characters (for example {" can be a single token).
• Mask / logit masking — setting the logit of disallowed tokens to -infinity so their probability becomes exactly 0 after softmax.
• Grammar — a formal set of rules defining which strings are valid (a JSON schema, a regex, a programming language syntax).
• Regular expression (regex) — a pattern language describing a “regular” set of strings; can be compiled to an FSM.
• FSM / DFA (Finite-State Machine / Deterministic Finite Automaton) — a machine with a finite set of states and, for each state and input character, exactly one next state; it accepts a string if it ends in an accepting state.
• CFG (Context-Free Grammar) — a more powerful rule system that can express nesting (balanced brackets, recursive JSON) which an FSM cannot; needs a stack to run.
• JSON Schema — a JSON document describing the allowed shape of other JSON (keys, types, required fields, enums).
• JSON mode — a provider feature that forces syntactically valid JSON, without a specific schema.
• Tool / function calling — an API feature where the model emits a structured request to call a named function with arguments.
• Chat template — the model-specific recipe (shipped with its tokenizer) that renders a list of messages and tool definitions into one flat prompt string with delimiter tokens.
• Special token — a reserved vocabulary entry (for example <tool_call> or <|im_end|>) used as a structural marker and never produced by tokenising ordinary text.
• SFT (Supervised Fine-Tuning) — training on example input/output pairs so the model imitates the target outputs.
• RL (Reinforcement Learning) — training against a reward signal (for example “did the tool call solve the task”) instead of fixed target text.
• Tool-call parser — inference-server code that extracts the call block from generated text and returns it as a structured {name, arguments} object.
• Prefix (viable prefix) — a partial string that can still be extended into a valid complete string.
• Sampler — the component that picks the next token from the (masked) distribution; see Decoding parameters: temperature, top-p, top-k.
• Hallucination — a fluent, plausible but false or ungrounded output.
• Truncation (max_tokens / length stop) — generation stopped by the output-length limit rather than the model finishing.
• Canonical tokenization — the token split the tokenizer itself would produce for a string; the one the model saw almost always in training. Other splits of the same string are valid but rare.
• Token healing — backing up one token at a boundary and constraining the next token to start with the removed characters, so the model can choose its natural whole token.
• Chain rule (of probability) — the probability of a text is the product of each token’s probability given all earlier tokens: p(t1) * p(t2 | t1) * ....
• Scratchpad — free-form reasoning text the model writes before its final answer.

The problem

A model asked for JSON is only likely to produce JSON. It is a next-token predictor; one stray “Sure, here is your JSON:”, a missing comma or a trailing quote can break your parser. Prompting harder reduces, but does not eliminate, these failures.

The mechanism: mask, then sample

At every generation step the model produces logits over the whole vocabulary. Normal decoding samples from them (see Decoding parameters: temperature, top-p, top-k). Constrained decoding inserts one step before sampling:

1. model forward pass          -> logits z for every token
2. grammar state S (what has been emitted so far)
                               -> set A of tokens that can legally come next
3. mask: z_i <- z_i   if i in A
         z_i <- -inf  otherwise          (so p_i = exp(-inf) = 0)
4. softmax, temperature, top-p, etc. on the masked logits
5. sample a token, advance the grammar state S with it
6. repeat until the grammar reaches an accepting state (or max length)

Symbols: z_i is the logit of token i; A is the allowed set computed from the grammar; S is the grammar’s current position (an FSM state, or a stack for a CFG). Because -inf becomes probability exactly 0 after softmax, the model is structurally incapable of choosing a token outside A. It is not asked nicely; the option does not exist.

This composes with normal sampling: temperature and top-p still apply within the grammar-valid subset, and the surviving probabilities are renormalised.

Worked example: a tiny grammar

Goal: output exactly {"ok": true} or {"ok": false} (the space after the colon optional). The toy vocabulary has 13 tokens, deliberately including multi-character ones and junk the model would like to say: {", ok, ":, , true, false, }, Sure, ,, ", tr, ue, maybe. I made the stand-in logits random and boosted “Sure” by +6, so an unconstrained model would almost always start with “Sure”. The run below was produced by the code in the next concept:

text so far          allowed tokens                         chosen
''                   ['{"']                                 '{"'
'{"'                 ['ok']                                 'ok'
'{"ok'               ['":', '"']                            '":'
'{"ok":'             [' ', 'true', 'false', 'tr']           ' '
'{"ok": '            ['true', 'false', 'tr']                'false'
'{"ok": false'       ['}']                                  '}'

Even with “Sure” strongly preferred by the model, it is never allowed: at step 1 the only legal token is {". At step 4 the model picks among four tokens; “maybe”, “Sure” and “,” are masked.

Notice two details. tr is allowed because {"ok": tr is a viable prefix of {"ok": true; this is the token-boundary problem covered below. And " is allowed after {"ok because {"ok" is a prefix of {"ok":.

Efficiency

Naively recomputing A by testing all 100,000+ vocabulary tokens against the grammar at every step is slow. Production implementations precompile the grammar into a table (for an FSM: for each state, the set of tokens that keeps you inside the language), so each step is a lookup plus a mask. CFG-based engines keep a stack and cache aggressively; cost is typically a small per-token overhead compared with the model forward pass, but compiling a large schema the first time can add noticeable latency, which is why providers cache compiled grammars.

What it is

For regular grammars (regexes, flat schemas) the grammar is a DFA over characters. To constrain a token-level model, you ask for each state: “which vocabulary tokens, when their characters are fed through the DFA from this state, never fall off the machine?” Those tokens form the allowed set for that state.

Diagram for {"ok": true|false}

 s0 --{--> s1 --"--> s2 --o--> s3 --k--> s4 --"--> s5 --:--> s6
                                                               |
                                    optional single space ' ' -+
                                                               v
                                                              s7 --t,r,u,e--> s8 --}--> s10 (accept)
                                                               |                          ^
                                                               +--f,a,l,s,e--> s9 --}-----+

The idea is that each state lists the characters that keep the string a valid prefix. A table form is easier to read:

state (text so far)	legal next characters
`` (start)	{
{	"
{"	o
{"ok	"
{"ok"	:
{"ok":	or t or f
{"ok":	t or f
{"ok": true or {"ok": false	}
complete	(accepting: stop)

Token-level allowed sets are derived from this by checking, for each vocabulary token, whether all its characters can be consumed in sequence.

Runnable toy implementation

import math, random, re

vocab = ['{"', 'ok', '":', ' ', 'true', 'false', '}', 'Sure', ',', '"', 'tr', 'ue', 'maybe']
PATTERN = re.compile(r'\{"ok": ?(true|false)\}')
LANGUAGE = ['{"ok":true}', '{"ok": true}', '{"ok":false}', '{"ok": false}']

def is_viable_prefix(s):
    return any(w.startswith(s) for w in LANGUAGE)

def allowed_tokens(text):
    return [i for i, t in enumerate(vocab) if is_viable_prefix(text + t)]

def masked_logits(z, text):
    ok = set(allowed_tokens(text))
    return [x if i in ok else -math.inf for i, x in enumerate(z)]

def softmax(z):
    m = max(z); e = [math.exp(x - m) for x in z]; s = sum(e)
    return [x / s for x in e]

random.seed(1)
text = ""
while not (text and PATTERN.fullmatch(text)):
    z = [random.gauss(0, 2) for _ in vocab]       # stand-in for real model logits
    z[vocab.index("Sure")] += 6                    # the model "wants" to say Sure
    p = softmax(masked_logits(z, text))
    i = random.choices(range(len(vocab)), weights=p)[0]
    text += vocab[i]
print(text)
# {"ok": false}

Walk-through: is_viable_prefix answers “can this text still grow into a valid string?” by checking against the four valid outputs (a real engine uses a compiled automaton instead of a list). allowed_tokens tries each vocabulary token appended to the text so far. masked_logits replaces everything else with -inf. The loop is real decoding in miniature: model logits (faked here), mask, softmax, draw, append. The output is always one of the four valid strings no matter what the “model” prefers.

Context-free grammars and nesting

An FSM cannot match nested structure of unbounded depth ([[[...]]], objects within objects) because it has no memory of how many brackets are open. A CFG engine adds a stack: after { it pushes “expect }”, after a nested [ pushes “expect ]”, and the allowed set depends on the stack top. General JSON, programming-language syntax and recursive schemas need this. Many JSON Schemas can be flattened to FSMs when nesting depth is bounded or when the schema has no recursion; recursive schemas need a CFG.

approach	guarantees	does not guarantee	cost / setup
prompt only (“reply in JSON”)	nothing	validity, keys, types	free; fragile
JSON mode	output parses as JSON	your keys, types, required fields	cheap, broad, weakest
schema / CFG constrained	parses and matches the schema’s shape (required keys, enum values, types)	that values are true	needs engine support; grammar compile time
native tool calling	model emits a call to a named tool with arguments matching the tool’s schema	that the arguments are right, or that the model chose the right tool	designed API; often built on the same machinery

Example of the difference

You want {"sentiment": "positive" | "negative", "score": 0..1}.

• Prompt-only might return Here is the JSON: {"sentiment": "pos"....
• JSON mode returns valid JSON like {"mood": "happy"}: parseable, but wrong keys.
• Schema-constrained returns only a document with a sentiment that is one of the two enum strings and a numeric score.

A schema in the shape most APIs accept:

{
  "type": "object",
  "properties": {
    "sentiment": { "type": "string", "enum": ["positive", "negative"] },
    "score": { "type": "number" }
  },
  "required": ["sentiment", "score"],
  "additionalProperties": false
}

Note the grammar can express type and enum, but typically not numeric ranges such as 0 <= score <= 1 (support depends on the engine); validate that in code afterwards.

Why prompt-only is not enough in production

Instructing the model to “respond only in JSON matching this schema” works most of the time in a demo, but is measurably less reliable at production volume: the model occasionally wraps the JSON in explanatory commentary, produces syntactically malformed output, or drifts from the requested schema, especially on adversarial or unusual inputs. Provider-enforced modes (structured output, JSON mode, function/tool calling) move the constraint from a request in the prompt text down to the token-sampling step itself, which is meaningfully more reliable — though, depending on the provider and feature, still not an absolute guarantee (refusals, truncation at the token limit, and unsupported schema features can all still produce unusable output).

Tool calling

With native tool calling the model’s “output” is: which tool, plus arguments in the tool’s schema. Framing differs (an action request, not a free-form document) but the guarantee is the same kind: the arguments conform in shape. Check exact support and API shape in your provider’s documentation, since the field names differ between vendors.

When to use which

For “I need this specific shape reliably”, use schema-constrained decoding or native tool calling. For “I only need it to parse”, JSON mode is enough. If your stack offers neither, fall back to prompting plus validation and retries. Whichever you choose, keep defensive parsing with a retry-on-parse-failure path as a safety net: no current approach is perfectly failure-proof (see the defensive-parsing concept below).

The model has no “call a function” operation. It has one operation: predict the next token. A tool call is therefore a pattern of tokens the model was trained to emit in the right situation, plus serving code that recognises the pattern and turns it back into a structured object. (What your application then does with that object is the agent loop; see the AI agents topic, What actually happens when an LLM “calls a tool”.) This concept covers the part inside the model and its inference server.

The pipeline, end to end

tool definitions (JSON Schema)  +  message list
              │
              ▼
   chat template renders ONE flat string          ← tools become plain prompt text
              │
              ▼
   tokenizer → token ids (incl. special tokens)
              │
              ▼
   model generates, token by token                ← the "decision" is just token probabilities
        ├─ ordinary text tokens     → a prose answer
        └─ <tool_call> {...} </tool_call>   → a call
              │
              ▼
   inference server parser: text → {name, arguments}   ← structured object for the API response
              │
              ▼
   (your code runs the tool; the result is appended as a "tool" message and the cycle repeats)

1. Tool definitions are rendered into the prompt

The tools list you send is not a separate channel. The model’s chat template (see System, user, and assistant roles at the token level) inserts the names, descriptions and JSON Schemas into the prompt as text, usually in or right after the system turn. You can see exactly what the model reads with an open-weights model:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Current temperature in Celsius for a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]
messages = [{"role": "user", "content": "Is it warmer in Paris or Oslo?"}]

print(tok.apply_chat_template(messages, tools=tools, tokenize=False, add_generation_prompt=True))

An abbreviated version of what it prints (the format is model-specific; this one is the Hermes / Qwen style):

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.

# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "get_weather", ...}}
</tools>

For each function call, return a json object with function name and arguments
within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call><|im_end|>
<|im_start|>user
Is it warmer in Paris or Oslo?<|im_end|>
<|im_start|>assistant

The prompt ends at the assistant’s turn, and the model continues from there. Consequences worth knowing: tool definitions cost input tokens on every request, the descriptions are prompt text the model reads (a vague description is a vague instruction), and a template that does not know about tools (or the wrong template for the model) makes the model see no tools at all, so it answers in prose or invents a format.

2. The model is trained to emit the call format

Instruction-tuned models are not prompted into tool use by the text above alone; the format in the prompt matches what post-training taught them (see Training and post-training):

• Supervised fine-tuning (SFT) on conversations that include tool-call turns: given a request plus tool definitions, the target assistant text is <tool_call>{"name": ..., "arguments": ...}</tool_call>; given a tool result, the target is a final answer or the next call.
• Reinforcement learning (RL) on tasks where the reward depends on the tool outcome (correct tool, valid arguments, task solved), which sharpens when to call and which tool.
• Loss masking: the training loss covers the assistant’s tokens (including the call) but not the tool results, which are inputs the model cannot control.

What the model has actually learned is three conditional distributions over tokens:

decision	where it lives in generation
Call or answer?	The probability of the first token of the assistant turn: the call-opening token (<tool_call>) versus an ordinary text token.
Which tool?	The tokens after the opening: the name is copied from the definitions in context, conditioned on the user request and descriptions.
What arguments?	JSON tokens whose values are mostly copied or transformed from the user text, earlier tool results, or context, shaped by the parameter schema.

All three are probabilistic. A tool call is not a deterministic branch, which is why the same request can sometimes answer directly and sometimes call, and why sampling temperature matters for argument quality (see Decoding parameters: temperature, top-p, top-k).

3. Special tokens make the boundary detectable

Formats use special tokens (reserved vocabulary entries that never appear in normal text) or fixed marker strings for the call boundaries and the turn end, for example <tool_call>, </tool_call>, <|im_end|>, or a dedicated “tool call” role/end token. Two reasons:

• The model can learn a clean, unambiguous signal for “a call starts here”, instead of having to judge from ordinary prose.
• The serving layer can detect it cheaply: when the model ends its turn right after a call block, generation stops (a stop token or sequence), and nothing further is generated until the tool result arrives.

4. The server parses the text back into a structure

The API response with a tool_calls field is built by the inference server, not emitted by the model. A tool-call parser, specific to the model family, extracts the block between the markers, parses the JSON, and returns {name, arguments} with an id. This is why:

• self-hosted serving needs the right parser for the model (for example a --tool-call-parser option in vLLM, or a matching chat template in llama.cpp); the wrong one silently returns the call as plain text;
• streaming a tool call delivers argument fragments (partial JSON) that must be accumulated before they are usable;
• a malformed block (truncated by max_tokens, stray text, broken JSON) arrives as a parse failure or as plain text, not as a call.

5. Constraining the call with a grammar

Training makes malformed calls rare; it does not make them impossible. Many engines add constrained decoding (see How grammar-constrained decoding guarantees syntactically valid output) for the call itself, usually triggered only after the model emits the opening marker, so prose answers stay unconstrained:

• the tool name is restricted to the set of defined names;
• the arguments are restricted to the JSON Schema of that tool, so required keys and enum values are always valid.

The tool_choice setting is the same mechanism, driven by the caller:

tool_choice	what happens inside
auto	The model decides freely (the first-token distribution above).
required / “any tool”	The call opening is forced, for example by starting the assistant turn with it or by a grammar that allows only calls.
a specific tool	As above, with the name fixed to one tool.
none	Tools are not offered, or call tokens are masked, so only prose can be produced.

Pitfalls

• Too many or overlapping tools dilute attention and raise wrong-tool errors; descriptions compete for the same context (see Why tool-use reliability degrades in long agent runs).
• Shape is guaranteed at best, truth is not. A grammar can force city to be a string; it cannot force it to be a real city or the right one (see What constrained decoding does not guarantee — syntax vs semantics).
• Tool results are just more prompt text. The model reads a tool result with the same attention as everything else, so an injected instruction inside a result can steer it (see Indirect prompt injection and the confused-deputy problem for tool-using agents).
• Format mismatch with open-weights models (wrong chat template, wrong parser, a model not trained for tools) is the most common reason tool calling “doesn’t work” locally.

Grammars, regexes and JSON Schemas are written in characters. The model emits tokens: chunks of one or more characters chosen by a tokenizer for compression reasons, not to line up with any grammar. A constrained decoder has to bridge those two levels at every step, and the mismatch is the source of most of the engineering difficulty (and a few quality problems).

The mismatch in one example

Target format: a phone number \d{3}-\d{4} (three digits, a dash, four digits). At the character level the question is trivial: position 0 must be a digit, position 3 must be -. At the token level the vocabulary might contain 5, 55, 555, -, -12, 7-, 1234 and thousands of other pieces. Which tokens are legal after 55 have been written?

from functools import lru_cache

PATTERN = ["D", "D", "D", "-", "D", "D", "D", "D"]          # \d{3}-\d{4}, one class per position

def char_ok(cls, ch):
    return ch.isdigit() if cls == "D" else ch == cls

def advance(state, token):
    """Feed the token's characters through the pattern one by one.
    Returns the new state (position), or None if any character is illegal."""
    for ch in token:
        if state >= len(PATTERN) or not char_ok(PATTERN[state], ch):
            return None
        state += 1
    return state

vocab = ["5", "55", "555", "-", "55-", "-1", "-12", "1234", "12", "34", "7-", "abc", " 555", "5551"]

for state, text in [(0, ""), (2, "55"), (3, "555")]:
    allowed = [t for t in vocab if advance(state, t) is not None]
    print(f"text={text!r:6} next char must be {PATTERN[state]!r:4} allowed tokens={allowed}")
# text=''     next char must be 'D'  allowed tokens=['5', '55', '555', '12', '34']
# text='55'   next char must be 'D'  allowed tokens=['5', '7-']
# text='555'  next char must be '-'  allowed tokens=['-', '-1', '-12']

@lru_cache(None)
def count_paths(state):
    if state == len(PATTERN):
        return 1
    return sum(count_paths(advance(state, t)) for t in vocab if advance(state, t) is not None)

print(count_paths(0))      # 395  distinct token sequences that spell a valid phone number

Walk-through of the output:

• State 0. The next character must be a digit, but the allowed set contains 5, 55, 555, 12, 34. Multi-character tokens are fine as long as every character fits. 1234 is rejected: its first three characters are digits but the fourth is 4 where the pattern needs -. 55- is also rejected here (it would put a dash at position 2).
• State 2 (55 written). 55 is rejected (position 3 needs -), but 7- is allowed: the 7 fills position 2 and the - fills position 3 in one token. The token straddles two grammar positions. A naive “next character must be a digit” rule would have wrongly forbidden it.
• State 3. Tokens that begin with the dash and continue (-1, -12) are allowed too, so the mask must test whole tokens, not first characters.
• 395 paths. With only 14 toy tokens there are already 395 different token sequences that produce a valid number.

What this means for the implementation

character-level grammar state  s   (e.g. position 2 in the pattern, or a stack for JSON)
              |
              |  for every vocabulary token t (100k+ of them):
              |     run t's characters through the grammar from state s
              |     legal only if ALL characters are accepted
              v
        allowed set A(s)   ->   mask logits   ->   sample   ->   advance s by the chosen token's characters

Naively that is a 100k-token scan per generated token. Engines therefore precompute an index: for a regex-compiled DFA, a table state -> allowed token ids and (state, token) -> next state, built once per grammar (this is the compile cost mentioned earlier). For CFGs (JSON with nesting) the allowed set depends on the stack too, so engines cache by grammar position and test only candidates that survive cheap pre-filters.

Consequence 1 - the same string has many tokenizations

true can be one token true, or tr + ue, or t + r + ue, and so on. The grammar accepts all of them because it only looks at characters. During training the model almost always saw the canonical tokenization (the one the tokenizer itself would produce, true). Non-canonical splits are rare in training data, so the model assigns them low probability and its behaviour after an unusual split is less reliable.

When does the constraint push the model onto a non-canonical split? When the preferred token is not allowed at that position. Example: your grammar says the value must be exactly "yes" or "no" followed by a closing quote. If the model’s favourite token is yes" (one token containing the quote) and the vocabulary has it, fine. If the grammar was written so that the closing quote must be a separate step and yes" is judged illegal, the model’s top choice disappears and it must write yes then ": a path it has seen less often. The output is still valid, but the probability mass is being squeezed through low-probability routes.

Consequence 2 - the prefix problem (“token healing”)

Suppose your prompt (or the already generated text) ends in the middle of what the tokenizer would treat as one token: a prompt ending in http: when the training data almost always tokenized :// as a unit. The model must now continue from a boundary it rarely saw. The usual fix, token healing, backs up one token and constrains the next token to start with the removed characters, so the model can pick the natural whole token. Constrained decoding engines use the same trick at the start of a constrained span, for the same reason.

Consequence 3 - whitespace and formatting fight the model’s habits

If the grammar demands compact JSON ({"a":1}) but the model learned to write {"a": 1} (space after the colon), the model’s favourite token is masked at every key. Probability renormalises onto less natural tokens each time. Quality can dip slightly and output can become odd (excess whitespace is a classic symptom). Practical fixes: allow optional whitespace in the grammar, match the format the model naturally produces, and put an example of the exact format in the prompt so the model’s preferences line up with what the grammar allows.

Consequence 4 - distortion from renormalising

Masking removes probability mass and the rest is renormalised. With illustrative probabilities for the next token: true 0.70, tr 0.01, True 0.25, yes 0.04, and a grammar that allows only true and tr: the survivors have mass 0.71, so true becomes 0.70 / 0.71 = 98.6% and tr becomes 1.4%. That is as intended. But if the model’s high-probability tokens were all masked and the survivors have tiny probabilities (say 0.0001 total), renormalisation inflates them to 100%: the model is being forced down a path it considers extremely unlikely, and the following tokens are conditioned on something unnatural. Low total surviving mass at a step is a useful warning signal: it means the grammar and the model disagree strongly there.

When to use which approach

situation	approach
provider offers schema-constrained output	use it; the provider already handles tokenization correctly
you run an open model locally	use an engine with built-in grammar support rather than writing your own masking loop
writing your own grammar	allow the whitespace and quoting style the model naturally produces; test with real prompts and inspect the surviving mass
tokens straddling grammar boundaries cause odd outputs	make the grammar token-aware or loosen formatting rules

It guarantees the output parses and matches the schema’s shape (right keys, right types, right structure). It does not guarantee the values are correct, complete or meaningful. A grammar-constrained extraction can confidently produce {"invoice_total": 4213.50} in perfectly valid JSON while that number is entirely fabricated, because the grammar constrains syntax, not semantics: it cannot compare the number against the source document.

Example

Source invoice text: “Total due: EUR 1,243.50”. Constrained output:

{"invoice_total": 4213.50, "currency": "EUR"}

Valid, schema-conforming, wrong (digits transposed). A parser-exception rate of 0% tells you nothing about this 1-in-N error. Constrained decoding can make the problem less visible because bad outputs no longer crash anything.

Mitigations

• Ask for evidence: a field source_quote containing the exact text the value came from, then verify with a substring check in code.
• Cross-field checks: line items must sum to the total; dates must be ordered; currency must be in a known set.
• Spot-check evals on a labelled sample, tracking field-level accuracy rather than “valid JSON rate”.
• Allow an explicit null or "unknown" option, otherwise a required field forces the model to invent a value.

That last point is a frequent trap: marking a field required with no way to say “not present” turns missing information into hallucination.

A model “thinks” by writing tokens (see Reasoning models and test-time compute): each intermediate token becomes context for later ones. A schema that forces the very first tokens to be the final answer removes that working space. This concept explains the mechanism with the chain rule, shows the two field orders side by side, and gives three mitigations.

The mechanism in terms of probabilities

The model generates a text by multiplying conditional probabilities, one token at a time (the chain rule; see Model architecture: vectors, encoders, decoders). Compare two ways of producing the same final value A for a question Q that needs several steps:

answer first:      p(A | Q)                      the model must guess A in ONE step, from Q alone
reason first:      p(R | Q) * p(A | Q, R)        R = the written-out working; A is then conditioned on R

Symbols: Q is the prompt/question, R is the reasoning text (many tokens), A is the answer tokens. In the first form there is a fixed, small amount of computation (one forward pass per answer token) between reading Q and committing to A. In the second, R provides many extra forward passes and stores intermediate results where later tokens can attend to them. For a question like “17 x 24 + 9” the answer-first model has to produce the digits 4 1 7 essentially from pattern recall; the reason-first model writes 17 x 24 = 408, 408 + 9 = 417 and each step is easy.

Same task, two schemas

{
  "type": "object",
  "properties": {
    "answer":    { "type": "number" },
    "reasoning": { "type": "string" }
  },
  "required": ["answer", "reasoning"]
}

With this property order, the grammar (and typically the generated JSON, because most engines emit properties in schema order) forces {"answer": first. The model must commit to a number before writing any working. The reasoning field that follows becomes a rationalisation of an answer already chosen, not the cause of it.

{
  "type": "object",
  "properties": {
    "reasoning": { "type": "string" },
    "answer":    { "type": "number" }
  },
  "required": ["reasoning", "answer"]
}

Now the output begins {"reasoning": "17 x 24 = ..." and the number comes last, conditioned on the working. The grammar allows free text in a string field, so nothing about the constraint prevents thinking; only the order changes. Whether an engine honours property order depends on the implementation (some reorder or alphabetise keys), so check your provider’s documentation and test it.

Worked illustration with numbers

Illustrative figures (not a measured benchmark): suppose on a set of 100 two-step arithmetic questions a model answers 62 correctly when it must answer immediately and 91 correctly when it writes working first. Forcing answer first throws away the gap (29 questions). The size of the gap varies enormously by task: for a simple lookup (“what is the capital of France”) there is no gap, so schema order does not matter; for multi-step logic, maths, planning, or classification that needs weighing evidence, it can be large. Measure it on your own task by running the same cases with both orders and comparing accuracy.

Mitigations

1. Reasoning field first. Put a reasoning (or analysis, scratchpad) string property before the answer property in the schema. Cheapest change; you keep a single call and a single output to parse. Add a one-line description such as “Work through the problem step by step before answering”.
2. Two-phase generation. Call 1 unconstrained: ask for working and a conclusion in free text. Call 2 (or a continuation in the same call) constrained: convert that text into the schema.
3. Use the model’s built-in reasoning mode (extended thinking, where available) and constrain only the final response. The thinking happens in its own channel, before the constrained output begins, so the schema never interferes.

import json

def extract_with_thinking(generate, generate_json, question: str) -> dict:
    """generate(prompt) -> free text. generate_json(prompt, schema) -> constrained JSON string.
    Both are placeholders for your provider's calls."""
    notes = generate(
        f"{question}\n\nWork through this step by step, then state your conclusion on a final line."
    )                                                      # phase 1: unconstrained deliberation
    schema = {"type": "object",
              "properties": {"answer": {"type": "number"}},
              "required": ["answer"], "additionalProperties": False}
    raw = generate_json(
        f"Question: {question}\nWorking:\n{notes}\n\nReturn only the final answer.", schema
    )                                                      # phase 2: constrained, trivial to get right
    return json.loads(raw)

# phase 1 text: "17 * 24 = 17*20 + 17*4 = 340 + 68 = 408. 408 + 9 = 417. Conclusion: 417"
# phase 2 json: {"answer": 417}

Walk-through: phase 1 gives the model unlimited room to compute; phase 2 sees the finished working, so the constrained call is a trivial copy. The trade-off is one extra call (latency and cost) and a longer prompt in phase 2.

Choosing

situation	choice
extraction of values stated in the text (no inference)	answer-only schema is fine
classification needing judgement, maths, multi-step reasoning	reasoning-first field, or reasoning mode + constrained answer
latency or cost is tight	reasoning-first field in one call (costs only the extra output tokens)
you also need to audit why	keep the reasoning field and store it, but remember it may not be a faithful explanation (see Reasoning models and test-time compute)

Pitfalls

• Reasoning inside a JSON string must itself be valid JSON text (escaped quotes and newlines); the grammar handles this, but very long reasoning strings can hit max_tokens and truncate the document before the answer field is written. Leave generous headroom and check the stop reason.
• Do not leave reasoning after answer and expect the same benefit.
• Unconstrained free text is also where errors hide: validate the final answer with code where you can.

Validate after generation

Treat generation as unconstrained, then accept or reject the whole output. One wrong token anywhere can invalidate the response and force a full retry (another full generation’s cost and latency), with no guarantee that the retry succeeds, especially for a model that is not reliable at the format.

import json

def call_with_retry(generate, validate, attempts=3):
    last_err = None
    for _ in range(attempts):
        raw = generate()
        try:
            return validate(json.loads(raw))
        except (json.JSONDecodeError, ValueError, KeyError) as e:
            last_err = e                      # could feed e back to the model as a hint
    raise RuntimeError(f"gave up: {last_err}")

If each attempt has probability q of being valid, expected attempts are 1/q; with q = 0.8 that is 1.25 generations on average, but the unlucky tail costs 2-3 and latency doubles on those requests.

Constrain during generation

Intervene while generating: the invalid path is never taken, so there is nothing to reject and no wasted retry. Every generation succeeds syntactically by construction.

Trade-offs

	after-the-fact	constrained
needs logits access / grammar-aware sampler	no	yes
wasted retries	yes	none (syntax)
per-token overhead	none	small mask cost; grammar compile on first use
can distort content	no	occasionally: forced away from its preferred path, can give contorted values or odd token splits
semantic correctness	not covered	not covered

Use constrained decoding when your stack supports it and the shape is fixed; keep validation as well, because the two address different failure classes.

Several genuine gaps remain even with hard syntactic guarantees:

• Not every provider/model supports true constrained decoding; some only offer best-effort JSON mode, so validity itself is not always guaranteed.
• Values can still be semantically wrong (see above); range checks, required-field presence beyond bare schema validity and cross-field consistency catch classes of error the grammar cannot express.
• Truncation. If the response is cut off by a token limit mid-structure, the output is an incomplete prefix, not a valid document. The parser must detect that distinctly from “invalid JSON”.
• Multi-step pipelines compose. A value extracted correctly in one call can fail validation once combined with data from another source, which the generation-time grammar cannot see.
• Schema feature gaps. Engines often support only a subset of JSON Schema (for example, limited handling of numeric bounds, string patterns or deep recursion); what the schema says and what is enforced can differ.

A defensive parser

import json

def parse_extraction(raw: str, stop_reason: str) -> dict:
    if stop_reason == "max_tokens":              # name of this field varies by provider
        raise ValueError("truncated: output hit the length limit")
    data = json.loads(raw)                        # may still raise on non-constrained paths
    total = data["total"]
    if not (0 <= total < 10_000_000):
        raise ValueError(f"total out of range: {total}")
    return data

print(parse_extraction('{"total": 4213.5}', "end_turn"))   # {'total': 4213.5}
try:
    parse_extraction('{"total": 42', "max_tokens")
except ValueError as e:
    print(e)                                               # truncated: output hit the length limit

Walk-through: truncation is checked first, from the stop reason rather than by trying to parse, because a half-written document might coincidentally parse as something else. Then the structural parse, then semantic range checks the schema cannot express.

Error paths to design deliberately: retry with a corrective message (“the previous reply was cut off; shorter output, same schema”), fall back to a stricter or more capable mode, or surface the failure rather than silently proceeding on bad data. Constrained decoding raises the floor dramatically but does not remove the need for the safety net.