Structured Outputs Are Not an Evaluation Strategy

Structured output support was the single biggest quality-of-life improvement in building on language models. Parsing failures went from a daily annoyance to a non-event, and a whole category of defensive code disappeared.

It also quietly convinced a lot of teams that they had solved correctness, and they had not.

What the guarantee actually covers

Constrained decoding restricts the token sampler at each step to tokens that keep the output a valid prefix of the schema. If the schema says the next thing must be a " or a }, no other token can be sampled.

That gives you a strong and narrow guarantee: the output will parse, and it will match the schema’s shape and types.

It does not give you:

A value that is true.
A value that is present in the source document.
A consistent value across two runs of the same input.
A sensible choice among enum members when none of them fits.

The last one is the most under-appreciated. If you constrain a field to "positive" | "negative" | "neutral" and the correct answer is “the document does not discuss this”, the model will pick one of the three. It has no other option - you removed it.

The failure mode this creates

Python

class Extraction(BaseModel):
    invoice_total: float
    currency: Literal["USD", "EUR", "GBP"]
    due_date: date

This schema will never fail to parse. It will also, on a document with no total, return a plausible float. On an invoice in Swiss francs it will return one of three wrong currencies. On a document with two dates it will pick one with no signal about which.

Every one of those errors flows downstream as a well-typed, confidently-shaped value. Schema validation catches malformed data. It has nothing to say about wrong data, and wrong data that validates is more dangerous than data that does not.

A parse error is a bug that announces itself. A hallucinated field that satisfies your schema is a bug that waits for the quarterly close.

Make absence expressible

The cheapest improvement to most extraction schemas is to allow the model to decline.

Python

class Extraction(BaseModel):
    invoice_total: float | None
    currency: str | None          # ISO 4217, not an enum of three guesses
    due_date: date | None
    fields_not_found: list[str]

Two changes, both small. Optional fields let the model represent “not in the document” instead of inventing. An explicit fields_not_found list turns absence into a positive signal you can measure, rather than a null you cannot distinguish from a parsing gap.

For enums, keep them only where the domain is genuinely closed, and always include an escape hatch member - "other" or "unknown" - with instructions on when to use it.

Evaluate the values, on real data

The evaluation that matters compares extracted values against a labelled set of real documents. There is no way around building one, and it is smaller than people expect: 150-300 examples spanning your actual document distribution is enough to detect the differences that matter.

Three metrics carry most of the signal:

Metric	Question it answers
Field accuracy	When a value was extracted, was it right?
Coverage	How often was a present field found at all?
False extraction rate	How often was a value produced for an absent field?

That third row is the one structured outputs make worse and the one teams almost never measure. Track it separately or it disappears into an accuracy average that looks fine.

Python

def score(pred: Extraction, gold: Extraction, fields: list[str]) -> dict:
    correct = absent_correct = hallucinated = missed = 0

    for field in fields:
        p, g = getattr(pred, field), getattr(gold, field)
        if g is None:
            absent_correct += p is None
            hallucinated += p is not None
        elif p is None:
            missed += 1
        else:
            correct += p == g

    return {
        "correct": correct,
        "missed": missed,
        "hallucinated": hallucinated,
        "absent_correct": absent_correct,
    }

Four counters, no framework. The point is that hallucinations and misses are counted separately, because the fixes are opposite: hallucinations want a stricter prompt and optional fields, misses want better retrieval or a larger context window.

Ask for the evidence

The single most effective schema change for auditability is a quote field alongside each extracted value.

Python

class Field(BaseModel):
    value: str | None
    source_quote: str | None   # verbatim span from the document

Then verify mechanically: does source_quote appear in the input document, as a substring? That check is deterministic, costs nothing, and catches a large fraction of fabricated values without a human in the loop. When the quote is real and the value disagrees with it, you have found a reasoning error rather than a retrieval one - a genuinely useful distinction when you are deciding what to fix.

Where this leaves structured outputs

Use them. They remove an entire class of engineering pain, and there is no reason to hand-roll JSON repair in 2026.

Just be precise about what they bought you. Constrained decoding is a parser guarantee. Correctness is still an evaluation problem, and it is still yours.