Lab 3: Trust but Verify — an LLM Output Verification Harness

Large language models produce output that looks authoritative and is sometimes quietly wrong — a total that does not add up, a field silently dropped, a response that is chatty prose where you asked for clean JSON. As AI moves into pipelines, the scarce skill is no longer generating text but checking it, automatically and at scale. In this lab you build a verification harness that catches broken LLM-style outputs before they reach production. No API key required: you verify against realistic fixtures that stand in for a model’s responses.

CT concepts fused in this lab: verification thinking (deciding in advance what “correct” means and enforcing it independently of the generator), evaluation & debugging (turning failures into precise, actionable diagnoses), and data thinking (reasoning about structure, invariants, and the ways structured data goes wrong).

You need: Python 3.9+ and a terminal. Build verify.py step by step. Time: roughly 70 minutes.

Decide what “correct” means before you look

The cardinal rule of verification thinking: define correctness before you see the output, or the output will define it for you. Our scenario: a prompt asks a model to extract a structured invoice as JSON. Using data thinking, enumerate the properties a valid invoice must satisfy — regardless of which model produced it or what the numbers are:

It must parse as JSON at all.
It must contain the required fields: id, currency, line items, subtotal, tax, total.
Line items must be non-empty (an invoice with nothing on it is suspicious).
The subtotal must equal the sum of item quantities times prices.
The total must equal subtotal plus tax (internal arithmetic consistency).
Tax must not be negative (a business rule).

These are invariants — statements that must hold for every valid output. Notice they split into three kinds: structural (1–3), arithmetic (4–5), and business rules (6). That taxonomy is data thinking at work, and it will directly shape the code.

Create realistic fixtures

You cannot test a checker without things to check. Instead of calling a real model, hand-write fixtures that mimic what LLMs actually return — including the characteristic ways they break. Start verify.py:

import json

# Each string is a pretend LLM response to "extract this invoice as JSON".
# Some are subtly broken in ways real models break.
FIXTURES = [
    # 0: clean and correct
    '{"invoice_id": "INV-1001", "currency": "USD", "line_items": '
    '[{"desc": "Widget", "qty": 2, "unit_price": 10.0}, '
    '{"desc": "Gadget", "qty": 1, "unit_price": 25.0}], '
    '"subtotal": 45.0, "tax": 4.5, "total": 49.5}',
    # 1: total does not equal subtotal + tax (arithmetic slip)
    '{"invoice_id": "INV-1002", "currency": "USD", "line_items": '
    '[{"desc": "Widget", "qty": 3, "unit_price": 10.0}], '
    '"subtotal": 30.0, "tax": 3.0, "total": 30.0}',
    # 2: negative tax (business-rule violation)
    '{"invoice_id": "INV-1003", "currency": "EUR", "line_items": '
    '[{"desc": "Service", "qty": 1, "unit_price": 100.0}], '
    '"subtotal": 100.0, "tax": -10.0, "total": 90.0}',
    # 3: chatty preamble + empty line_items
    'Sure! Here is the invoice you asked for:\n'
    '{"invoice_id": "INV-1004", "currency": "USD", "line_items": [], '
    '"subtotal": 0.0, "tax": 0.0, "total": 0.0}',
    # 4: truncated JSON (missing closing brace)
    '{"invoice_id": "INV-1005", "currency": "USD", "line_items": '
    '[{"desc": "Widget", "qty": 2, "unit_price": 10.0}], '
    '"subtotal": 20.0, "tax": 2.0, "total": 22.0',
]

Predict now: how many of these five should pass all six invariants? Write your guess down — you will check it in a few steps.

Extract JSON robustly

Fixture 3 shows a real-world nuisance: models wrap JSON in prose. A naive json.loads on the whole string would fail even though valid JSON is present. Data thinking says separate finding the data from validating it. Add a robust extractor:

def extract_json(raw):
    """Pull the first complete JSON object out of a possibly chatty reply."""
    start = raw.find("{")
    if start == -1:
        raise ValueError("no JSON object found")
    depth = 0
    for i in range(start, len(raw)):
        if raw[i] == "{":
            depth += 1
        elif raw[i] == "}":
            depth -= 1
            if depth == 0:
                return json.loads(raw[start:i + 1])
    raise ValueError("unbalanced braces: JSON object not closed")

This walks braces to find the first balanced object — handling the prose preamble in fixture 3 and raising a clear error for the truncation in fixture 4. A vague failure is a debugging tax you pay later; a precise one is an investment.

Encode each invariant as a small function

Now translate the six invariants into code. The design pattern matters: each invariant is an independent function that returns an error message on failure and nothing on success. Independence means one broken check never hides another, and adding a new rule never touches existing ones.

def inv_has_required_fields(d):
    required = {"invoice_id", "currency", "line_items",
                "subtotal", "tax", "total"}
    missing = required - d.keys()
    if missing:
        return f"missing fields: {sorted(missing)}"

def inv_nonempty_items(d):
    if not d["line_items"]:
        return "line_items is empty"

def inv_subtotal_matches_items(d):
    calc = round(sum(li["qty"] * li["unit_price"]
                     for li in d["line_items"]), 2)
    if calc != round(d["subtotal"], 2):
        return f"subtotal {d['subtotal']} != sum of items {calc}"

def inv_total_is_subtotal_plus_tax(d):
    calc = round(d["subtotal"] + d["tax"], 2)
    if calc != round(d["total"], 2):
        return f"total {d['total']} != subtotal+tax {calc}"

def inv_tax_nonnegative(d):
    if d["tax"] < 0:
        return f"tax is negative: {d['tax']}"

INVARIANTS = [
    inv_has_required_fields,
    inv_nonempty_items,
    inv_subtotal_matches_items,
    inv_total_is_subtotal_plus_tax,
    inv_tax_nonnegative,
]

The round(..., 2) calls are not decoration. Floating-point arithmetic makes 0.1 + 0.2 == 0.3 evaluate to False, so a naive equality check would flag correct invoices as broken — a false alarm that trains people to ignore the harness. Anticipating how your data type actually behaves is data thinking, and forgetting it is the classic verification bug.

Wire up the harness

Run every fixture through parsing and then every invariant, collecting all failures rather than stopping at the first:

def check_one(raw):
    failures = []
    try:
        d = extract_json(raw)
    except ValueError as e:
        return ["PARSE: " + str(e)]         # can't check further
    for inv in INVARIANTS:
        try:
            msg = inv(d)
            if msg:
                failures.append(f"{inv.__name__}: {msg}")
        except (KeyError, TypeError) as e:
            failures.append(f"{inv.__name__}: crashed ({e})")
    return failures

def run_harness(fixtures):
    passed = 0
    for i, raw in enumerate(fixtures):
        failures = check_one(raw)
        if not failures:
            passed += 1
            print(f"[{i}] PASS")
        else:
            print(f"[{i}] FAIL")
            for f in failures:
                print(f"      - {f}")
    print(f"\n{passed}/{len(fixtures)} outputs passed all checks")

if __name__ == "__main__":
    run_harness(FIXTURES)

Run python3 verify.py. Expected output:

[0] PASS
[1] FAIL
      - inv_total_is_subtotal_plus_tax: total 30.0 != subtotal+tax 33.0
[2] FAIL
      - inv_tax_nonnegative: tax is negative: -10.0
[3] FAIL
      - inv_nonempty_items: line_items is empty
[4] FAIL
      - PARSE: unbalanced braces: JSON object not closed

1/5 outputs passed all checks

Compare against your earlier prediction. One passes; the other four fail for four different reasons — arithmetic, business rule, structure, and parse. Each message names the offending check and shows the numbers, so a human (or an upstream retry loop) knows exactly what went wrong. That precision is the difference between evaluation and mere complaint.

The verification pipeline, and why the check is independent

Here is the shape of what you built:

    flowchart TD
    A[LLM-style output] --> B[Extract JSON]
    B -->|parse ok| C[Run every invariant]
    B -->|parse fails| E[Report PARSE error]
    C --> D{Any failures}
    D -->|no| P[PASS - safe to use]
    D -->|yes| F[Report each failure with detail]
    F --> R[Retry, reject, or escalate to human]

The crucial architectural property: the harness never asks the model whether its own answer is correct. The invariants are computed independently from ground rules. This is the antidote to the agentic-AI failure mode where a system grades its own homework — see Agentic AI and CT. Verification only means something when the verifier does not share the generator’s blind spots.

Add golden cases and catch a regression

Invariants check properties. Golden cases check exact expected values for known inputs — the second pillar of a verification suite. Add one and confirm it passes:

GOLDEN = [
    ('{"invoice_id": "G-1", "currency": "USD", "line_items": '
     '[{"desc": "A", "qty": 1, "unit_price": 5.0}], '
     '"subtotal": 5.0, "tax": 0.5, "total": 5.5}',
     {"invoice_id": "G-1", "total": 5.5}),
]

def check_golden(golden):
    for raw, expected in golden:
        d = extract_json(raw)
        for k, v in expected.items():
            assert d.get(k) == v, \
                f"golden mismatch on {k}: got {d.get(k)}, want {v}"
    print(f"golden: {len(golden)}/{len(golden)} matched")

Now do the exercise that makes verification real. Suppose a teammate “optimizes” extract_json to json.loads(raw[raw.find('{'):]) — slicing from the first brace to the end. Predict what breaks, make the change, and re-run. Fixture 3 (prose + trailing content is fine here) may survive, but any fixture with text after the JSON, and the truncated fixture 4, now fail differently — and your golden case plus invariants catch it instantly. A verification suite is not a one-time gate; it is a regression net that lets you change code without fear, because it re-checks everything every run. That is the same measure-diagnose-rerun loop as Lab 1, now aimed at AI output.

What you actually practiced

You defined correctness before generation, expressed it as independent invariants and golden cases, and built a checker that turns opaque failures into precise diagnoses — the core of verification thinking in the AI era. You handled the messy reality of model output (prose wrappers, truncation, floating-point traps) with data thinking, and you closed the loop with the evaluation and debugging discipline of reproducible, always-on checks. This harness is a toy, but its structure — independent verifier, layered invariants, golden regression cases, actionable messages — is exactly how production AI pipelines are kept trustworthy.

Going further

Schema first: express the structural invariants as a JSON Schema and validate against it, keeping only the arithmetic and business rules as custom functions.
Severity levels: distinguish ERROR (reject) from WARN (allow but log), since not every violation should block a pipeline.
Property-based fuzzing: generate hundreds of random valid invoices, confirm they all pass, then mutate one field each and confirm the right invariant fires — testing your tests.
Retry loop: when a check fails, feed the specific error back into a (real or simulated) regeneration step and re-verify, mirroring how agentic systems self-correct.
Cross-check with a second model: for claims no invariant can settle, compare two independent generations and flag disagreement — verification by consensus, with all its caveats.

Lab 2: Simulate an Epidemic