The value of the harness

A chat window is a harness, and it is the minimal one: a text box, a transcript, and the model, with nothing read before generation, nothing checked after it, and nothing kept between sessions. The same model dropped into a production system, one where every run reads written rules before generating, compares its work against graded examples, passes automated checks, and moves through a staged pipeline, produces work of a different class. In the chat window the model returns a draft that has to be steered, corrected, and checked by hand. In the production system it returns finished work. The intelligence is identical in both cases, so the whole difference in output is the value of the harness, made visible.

The gap has a mechanical explanation. Anyone who has worked with a language model has told it to think harder, and the model complies in the only way it can, by producing text that sounds like harder thinking: longer sentences, more qualifications, a more deliberate tone. The instruction buys a style, because a style is the only thing an instruction can buy in a bare harness. No dial inside the model maps “think harder” to more computation, so the words purchase the register of effort rather than the effort. The dial exists one level up. A pipeline can route a hard problem through more passes, more parallel candidates, and stricter checks, which is the extra computation the instruction was reaching for.

The bare harness has a second limit: nothing in it persists. A chat is one context window with no memory across sessions, so a correction given there improves the draft on screen, and then the window closes and the correction is gone. Both limits point the same way. A chat honors a request once, as a style, for one reply. A harness enforces the same demand on every run, as a mechanism, indefinitely.

What a harness is

In our systems the harness is concretely a codebase: the files, checks, examples, and pipeline code that exist between model calls and persist while the model inside is swapped per call. An effective harness, in our experience, has four parts, and the parts only work when they are versioned together as one unit.

• A skill file is the written rules of the craft, and every run reads it before generating anything.
• A golden set is a library of past outputs that were accepted, stored with the grades and the reasons, and new work is compared against it.
• A verifier is a set of automated checks that fail a draft when it violates a rule, and it runs without a human present.
• A rejection ledger is the record of failed outputs with the reason each one failed, and it is where the next rule comes from.

Around the four sits the pipeline: staged passes, checks between the stages, and a rule for how many parallel attempts get reduced to one accepted result.

The four pieces reference each other, which is why they version as one unit. A golden set without its verifier drifts, because nothing fails the examples that stop deserving their grades. A verifier without a ledger stops growing, because the record of what failed is where new checks come from. A rule without the example that motivated it gets argued with, by humans and by models alike. A change to any piece updates the others.

Written checks sound like evals, and an eval suite is one of the four pieces. The difference is where corrections land. Most eval practice grades the model and leaves the grader’s reasoning in a spreadsheet or a chat thread. When the four pieces are versioned together, every correction lands in an object that every future run must read, so nothing the operator learns evaporates.

The agent body describes an agent as a stateless mind visiting a persistent body, and the harness is the load-bearing part of the body: the part that carries what the operation has learned. Minds are rented from vendors and swapped without ceremony. The harness stays, and whatever the system learns is either written into it or lost.

What a release does

A model release is intelligence arriving from outside into whatever harness exists when it lands. In a bare harness the gain is mostly wasted: the smarter model re-derives context it was never given, repeats failures nobody recorded, and produces variance nobody can review. In a built harness the gain converts: the smarter model reads the same rules, passes the same checks, and clears work the previous model could not, often in a single pass. Work that needed steering under the old model becomes a one-shot process under the new one, and the reason is not the new model alone. The value was pre-baked into the harness while the old model ran, and the upgraded model collects it on arrival.

Not everything in the codebase is the value. A smarter model can regenerate scaffolding and glue from scratch, and it will, so the generic code depreciates along with the tuning. What no model can regenerate is the record of your selections against reality: which outputs you accepted, which you rejected and for what reason, which checks encode constraints of your actual domain. That part of the harness is data about judgment rather than code, it accumulates only through operation, and it is the part a release multiplies.

A natural experiment

During a client engagement we had temporary access to a frontier model. Partway through, the access was revoked, and some weeks later it came back. The revocation turned out to be a controlled experiment on where the value of the work lived.

While the access was gone, the work continued on weaker models, and the effort went into the harness. The frontier model’s earlier outputs were graded into a golden set. The patterns that made those outputs good were extracted into written rules. The checks that would have caught the weaker models’ failures were built and wired into the pipeline. When the access returned, the same model, no smarter than before, cleared the entire accumulated backlog almost immediately, at roughly thirty percent of the usage it had needed earlier, because it now flowed through the pipeline that had been built in its absence.

The model was identical before and after, so the only variable that changed was the harness. Before the revocation, a strong model in a thin harness produced good outputs slowly, each one hand-steered by the operator. After it, the same model in a built harness finished the backlog in one pass, and prompting effort went down rather than up, because the pipeline reads the rules so the operator does not restate them. A returning model is, from the harness’s side, exactly what a release is: intelligence arriving from outside into structure that was ready for it. Every future release replays this experiment with a stronger model in the returning role.

The harness could only be built because one real deliverable had already shipped: the graded examples had to come from work that had made contact with a real recipient, because a golden set graded against imagined standards records taste nobody has tested. A blind comparison we ran on the same class of system points the same way, and What compounds in an agent system reports it in full: the model mattered only where the tooling stopped.

Compiling judgment

The corrections a person gives in chat are the most valuable data they produce and the least preserved, because by default they die with the context window. Telling a model in chat to stop using a phrase fixes the draft in front of you and nothing else. Writing the same instruction into the skill file that every run must read, and storing the rejected draft in the ledger with the reason attached, fixes every draft produced after that day. The correction is identical in both cases. What differs is its half-life, which goes from one call to permanent when the correction lands in the versioned unit instead of the conversation.

The same move works one level higher. When outputs are wrong, the instinct is to argue with each one in chat, and the productive response is to change the rule, the example, or the check that allowed the output, which closes the whole class of failure at once.

Run over months, the loop amounts to compiling your own judgment into an executable form. Each rejection enters the ledger with its reason, each correction becomes a rule, each accepted output joins the golden set, and the compiled judgment then runs in every parallel agent at once, on every future model, with nobody present.

The frontier is rented. What you distill from it is owned.

The obvious worry is that compiling your judgment automates you away. What gets compiled is the settled part of the judgment, the rules you no longer want to restate, and compiling them moves the review upward: the person spends their time only on what the harness cannot yet check, and each session at that level produces the next rule. The operator’s position in the loop is permanent even as its altitude rises, and the skill practiced there shifts from producing outputs to producing the harness.

Compute market fit

A practical test for a harness is to drive more compute into it and watch what comes out. An effective one converts more compute into more finished, accepted work, a property we call compute market fit. A bare harness fails the test: more runs produce more variance to hand-review rather than more accepted output, because nothing converges the candidates. Verifiers and golden sets are what turn parallel generation from a review burden into a search: candidates that fail the checks are discarded mechanically, candidates that pass are compared against the graded examples, and the human sees only the survivors. The property that absorbs a smarter model is the same property that absorbs more compute from the current one, because both are more intelligence flowing in, and the harness is what turns intelligence flowing in into work coming out.

Model releases will keep arriving, on a schedule nobody downstream controls, and each one will land in whatever harness exists at that moment. In a bare harness a release is a slightly better chat. In a built harness it is a payout: the backlog clears, the cost per accepted result drops, and work that needed supervision becomes one-shot. The work that pays is building the harness the next model will flow through.