Evaluation & Debugging
The four classical pillars produce a solution. This fifth pillar asks the questions that make the solution trustworthy: Does it work? How do I know? What do I do when it doesn’t? Historically the least glamorous pillar, evaluation has become the most valuable one — in a world where AI generates candidate solutions in seconds, the human bottleneck is judging them (a shift explored fully in Verification Thinking).
Testing: structured doubt
Testing is the discipline of actively trying to make your own solution fail before reality does. Its core asymmetry, famously stated by Dijkstra: testing can show the presence of bugs, never their absence. That does not make testing weak; it makes test selection an intellectual act. Good testers spend their limited shots where bugs hide:
- Happy path — the intended use works end to end. Necessary, and nowhere near sufficient.
- Edge cases — empty input, one item, maximum size, duplicates, zero, negative numbers, Unicode, the last day of February. Boundaries are where off-by-one thinking lives.
- Invalid input — the system should fail well: clear error, no corruption, no silence.
- Regression — every bug you ever fixed gets a test that would have caught it. Bugs cluster and bugs return.
- Properties — instead of specific examples, assert invariants over random inputs: “output list is always sorted,” “money in equals money out.” Property-based testing automates the generation of the cases you failed to imagine.
A practical habit: write down the expected result before running the test. If you look at the output first, your brain will helpfully explain why whatever appeared is correct — the confirmation bias that testing exists to defeat.
Debugging: hypothesis-driven science
Debugging is not staring at code until enlightenment strikes. Done well, it is the scientific method applied to a system that is telling you, through a failure, that your model of it is wrong somewhere.
The loop, adapted from Andreas Zeller’s systematic treatment:
Reproduce
Make the failure happen on demand, ideally in the smallest possible setting. A bug you cannot reproduce cannot be studied — capturing the triggering input is half the work. Shrink it: delete input and steps until removing anything more makes the failure vanish. The minimal case usually points straight at the cause.
Observe precisely
What exactly is wrong — wrong value, wrong order, missing output, crash? “It doesn’t work” is not an observation. “The total is 19.99 when the items sum to 20.00” is, and it already whispers floating point to a pattern-matching mind.
Hypothesize one cause
Form a single, specific, falsifiable guess: “the discount is applied twice when a coupon and a sale overlap.” Rank competing hypotheses by likelihood and cheapness of testing. The classic heuristic: suspect your newest change first, your own code second, the library third, the compiler approximately never.
Design an experiment
Find the observation that would distinguish this hypothesis from rivals: a log line, a breakpoint, a unit test on the suspect function, bisecting the commit history, or deleting half the input (binary search generalizes beautifully to debugging). One variable at a time — change two things and learn nothing.
Conclude and either fix or loop
If the experiment refutes the hypothesis, that is progress: one suspect eliminated, model updated, return to step 3. If it confirms, fix the cause, not the symptom — then re-run the reproduction and the full test suite, and add the regression test.
Evaluation: is it a good solution?
Beyond “does it pass tests,” mature evaluation weighs a solution on at least three axes — and makes the trade-offs explicit:
Correctness. Does it meet the specification on all valid inputs, including the boundaries? Is the specification itself right — solving the user’s actual problem? A perfectly implemented wrong spec is still a failure, just a well-engineered one.
Efficiency. Time, memory, money, energy. Anchor this in the complexity intuition from Algorithm Design: is the growth curve compatible with realistic input sizes, not just today’s demo data? Efficiency also includes human efficiency — a solution requiring an expert on call at 3 a.m. has a cost line.
Robustness. How does it behave off the happy path — malformed input, a dependency timing out, double the expected load, a user clicking twice? Robust systems fail loudly, partially, and recoverably rather than silently, totally, and permanently. Ask “what happens when this assumption breaks?” for every assumption you can name.
These axes conflict. The most correct solution may be slow; the most robust one, complex. Evaluation is not scoring a single number — it is stating, honestly, which axis you privileged and why. “We chose the O(n²) approach because n is capped at 200 and the code is auditable” is an evaluation. “It works on my machine” is not.
Evaluation as a habit, not a phase
The strongest practitioners run this pillar continuously: they predict before executing, test each decomposed piece as it is built, and treat every surprise — even a pleasant one — as a signal that their mental model needs updating. That reflex, more than any syntax knowledge, is what transfers intact to the AI era, where the objects being evaluated are no longer only your own artifacts but a fast, confident, occasionally wrong collaborator’s. The techniques on this page apply verbatim; only the suspect list changes.