Evaluation and Verification: You Own Correctness

A deterministic system either passes its tests or it doesn't, and once it passes you stop thinking about it. An AI system has no such floor. The same prompt that nails one input quietly mangles the sixth, the model that was right last week regresses when the provider updates it underneath you, and none of this throws an error. The output that is wrong looks exactly as confident as the output that is right. That is the whole problem this lesson exists to solve: correctness is not a property the system guarantees, it is a property you have to measure and keep measuring, and if you don't own that, nobody does.

The first thing to internalize is that verification is not the cheap half of the work. It is the expensive half. Generating plausible output is nearly free now, while checking whether it is actually correct is slow, takes expertise, and is exactly the part a novice cannot do. The rough shape is familiar: a model gets you most of the way fast, and the last stretch, the debugging and the edge cases and the judgment about whether it is really right, is where it falls apart (the "70% problem," in Addy Osmani's framing). That last stretch is exactly what someone who couldn't have produced the output themselves has no way to supply. The practical consequence is uncomfortable. "This looks correct" should not relax you. On anything load-bearing it should make you verify harder, because the dangerous output is never the obviously-broken one, the kind you catch. It's the one that compiles, runs, passes the happy path, and is wrong somewhere you didn't look.

Most people's instinct when a prompt underperforms is to tweak the wording, eyeball one output, and ship. That isn't evaluation, it's fiddling, and it produces a prompt overfit to one lucky input. The reason eyeballing one output fails is sharper than "small sample." Changing things the model is sensitive to moves accuracy far more than intuition predicts, and the swings come from places that feel cosmetic. In the canonical study behind few-shot brittleness (Zhao et al., "Calibrate Before Use"), reordering the same few examples — identical content, different sequence — moved sentiment accuracy from 54%, barely better than a coin flip, to 93%, near the best the model could do. Nothing about the information changed. If a reshuffle you'd never think to test can swing a result by nearly forty points, then one good-looking output tells you almost nothing about whether the system is reliable. You need a fixed set of inputs you run every time.

The eval loop that replaces fiddling is small and unglamorous. Assemble a fixed test set of real inputs, and deliberately include the edge cases and the ambiguous ones, not three easy examples that everything passes. Change exactly one variable, the prompt or the model or the retrieval, never several at once. Run the whole set. Compare against the last run. Then ship or iterate. The discipline that makes it work is the one-variable rule: if you change the prompt and the model together and the score moves, you have learned nothing about which one did it. A small fixed set you actually run every time is worth more than an elaborate eval framework you set up once and never populate.

Once you are measuring, the next question is how to measure, and the operator move is to reach for the cheapest check that reliably catches a given error, not the most powerful one. There is a hierarchy here, cheapest and most reliable first. At the base sit deterministic assertions: is the JSON valid, are the required fields present, is it within length bounds, are the forbidden phrases absent. These cost almost nothing and catch a surprising share of failures, and you build them first. Above that, anywhere you can reframe the task as classification you get precision and recall, which are well-defined and stable. Above that, a narrow specialized model, like an entailment model that checks whether an answer actually follows from its source, runs cheaper and steadier than a general judge. Only at the top, for the genuinely subjective remainder that nothing simpler can measure, do you reach for an LLM-as-judge. The common mistake is to start at the top. Teams reach for an LLM judge first and skip the eighty percent they could have caught with a regex and a schema check.

There is one judge that everyone trusts and shouldn't: the model judging itself. "Ask the model to check its own work" is the most cited verification tip and one of the least reliable. Without some external signal of correctness, self-correction often degrades the output, because the model talks itself out of right answers as readily as wrong ones (Huang et al. find exactly this for reasoning, Stechly et al. for plans). The distinction that matters is whether the critic has access to something the generator didn't. A critic with test execution, a separate model, ground-truth data, or a human in the loop can genuinely improve the output, because it brings new information. "Reflect on your answer" with nothing new to reflect against is theater. It adds words and a feeling of rigor without adding correctness.

The measurement discipline has its own failure modes, and naming them is what keeps it honest. Generic benchmarks are the first trap. A high leaderboard score doesn't transfer to your task, because public benchmarks get contaminated when the test leaks into training, and gamed when labs optimize for the headline. The only number that tracks your reality is a small private eval set built from your own inputs. The deeper failure is human, not technical. When a system is usually right, reviewers stop genuinely reviewing, verification decays into rubber-stamping, and the rare wrong output sails through because the system is usually right and you've stopped looking hard. This is the same trap: once AI generates faster than you can internalize, your review quietly slides from mentally simulating the output into pattern-matching on whether it looks plausible, which is the worst position to verify from. The defense is behavioral, not a tool. Assume something is wrong on every load-bearing review, and verify the parts that matter most rather than skimming everything evenly. There is also a boundary condition worth stating: an eval set is worth building only when a task runs repeatedly or the cost of being wrong is real. For a one-off, low-stakes answer, a single careful read is the right amount of verification, and a twenty-example harness is over-engineering.