The Evaluation Harness Is the Product
Without a fixed, graded task set, every prompt change is a vibe. Building the harness first is the difference between an AI feature you can maintain and one you can only admire.
There is a moment in most applied-AI projects where the demo works, everyone is pleased, and the team quietly loses the ability to make changes. Someone edits a prompt to fix one bad output, three other cases silently regress, and nobody notices for a fortnight. The system has no test suite, so every change is a wager.
The fix is unglamorous and well understood in every other part of engineering: build the harness before the feature.
What a harness actually is
An evaluation harness has three parts. A fixed set of inputs. A grader that turns each output into a score. A runner that executes the set against a given configuration and reports the aggregate plus the diffs.
None of that is exotic. The difficulty is entirely in the first two.
Assembling the task set
The task set is not a random sample of production traffic. It is a deliberately constructed set that covers the behaviours you care about, weighted toward the ones that hurt when they break.
Start with roughly fifty cases and expect to reach two or three hundred. Include the obvious ones — typical inputs with clear correct answers. Then include the cases that actually decide whether the system is deployable: inputs where the correct response is to refuse, inputs that look answerable but are not, inputs containing instructions aimed at the model rather than at the task, and inputs from the edges of the distribution where the training data thins out.
Every production failure becomes a case. This is the single habit that separates systems that improve from systems that oscillate. A bug report that does not end with a new row in the task set will be re-fixed within the quarter.
Grading without fooling yourself
Grading is where teams either get honest signal or comforting noise.
Prefer deterministic grading wherever the output permits it. If the task produces structured data, compare fields. If it produces a classification, compare labels. If it must cite sources, check that the cited spans exist and support the statement. A surprising amount of what looks like open-ended generation can be reformulated to produce a checkable artefact, and doing so is often worth the constraint on its own.
For genuinely open-ended output, model-based grading is workable but requires care. Give the grader a rubric with concrete criteria rather than asking for a quality score. Have it grade one dimension at a time. Calibrate it against a few dozen human-graded examples before trusting it, and re-calibrate whenever the grading model changes. Treat the grader as a component with its own accuracy, because that is what it is.
Some things resist automatic grading entirely — tone, judgement, whether an answer is genuinely useful to the person who asked. Keep a small human-graded subset for those and run it less often. A weekly review of thirty outputs is more valuable than a metric nobody trusts.
Running it like a test suite
The harness has to be cheap enough to run on every change, or it will be run on none of them. Cache aggressively, parallelise, and keep a fast subset for the inner loop and the full set for merges.
Report the aggregate score, but treat the per-case diff as the primary output. Aggregates hide the failure mode that matters most in this work: a change that improves the average while breaking a category. The diff between two runs — which cases flipped, in which direction — is what tells you whether to keep the change.
Pin the model version explicitly and record it with every run. A provider updating a model behind a stable alias is a change to your system, and without a pinned version and a stored run history you will spend a week debugging your own code for a regression you did not cause.
Why this is the product
Once the harness exists, several things stop being arguments. Whether to switch models becomes a run, not a debate. Whether a prompt refactor is safe becomes a diff. Whether the system is good enough to ship becomes a threshold agreed in advance rather than a judgement made under deadline pressure.
The model is a dependency you do not control and cannot inspect. The harness is the part you own — the thing that makes the dependency safe to build on. Teams that internalise this ship AI features they can still change a year later. Teams that do not end up maintaining a prompt nobody dares touch.