Evaluation

Evaluation is how we judge whether an agent's work meets the task's requirements. The systems built so far can keep working across many model calls, but finishing a run does not establish that the task succeeded. A convincing answer may contain a mistake that the agent never noticed. To check the work, we need to decide what counts as success, obtain evidence about the result, and compare the two.

Criteria

A criterion is a condition we check to judge whether the agent did its job. It connects the task we gave the agent to a result we can inspect:

• A research agent's answer: the cited sources support the factual claims in the answer.
• A coding agent's bug fix: a test that reproduces the reported bug passes against the edited code, and existing tests still pass.
• An agent's document update: the saved document contains the requested change.

A task may need several criteria. The research answer may need to cover every part of the question as well as support its claims. A set of criteria is often called a rubric. The rubric tells the evaluator what to check while allowing different answers or approaches to satisfy the task. Keeping the results for each criterion makes it clear where the work fell short, which an overall score alone cannot explain.

Evidence

Once we know what to check, we need the material that can answer the question. To assess a factual claim in the research answer, the evaluator needs the claim and its supporting source. The agent's assurance that it checked the source adds little if we cannot inspect what the source says. This connects evaluation to the artifacts from earlier chapters: saved work gives a reviewer something to examine after the run ends.

Different requirements call for different evidence:

• The final answer: the claims, explanations, and formatting the reader receives.
• The resulting state: the actual file, document, or database record after the agent's action.
• The execution trace: the recorded operations, their results, and the time or resources they consumed.

If an agent says it fixed a bug, we can run tests against the changed files and record the results. Asking the model whether those tests would pass gives us its prediction. Running them gives us observations about the code.

The trace helps when the requirement concerns how the work was done. If a comparison must examine both company filings and product documentation, the recorded results can show whether the agent received relevant passages from both collections. A search call alone does not establish that coverage: we need to inspect what it returned. Several orders of searching and reading can satisfy the requirement, so the check should allow those alternatives unless the order itself matters.

Tests and validators

When a requirement has a precise rule, ordinary code can check it. A validator is a function that checks whether data follows such a rule. For citations, we can look up each source identifier in the stored sources. The example below receives a claim with a source identifier and a map of available sources. Its result names the condition checked and the claim it applies to.

function checkCitation(claim, sources) {
  const exists = sources.has(claim.sourceId)

  return {
    criterion: "Citation points to a stored source",
    result: exists ? "pass" : "fail",
    location: claim.id,
    evidence: exists ? [claim.sourceId] : [],
  }
}

This check can pass while the claim is wrong. A research answer might say “Product B is available now,” while the stored source says “Product B is scheduled for release next month.” The lookup correctly establishes that the source exists. Checking whether it supports the claim requires reading what the passage means. This is a common limit of evaluation: a check can work exactly as intended while leaving an important requirement untested.

Model-based review

A language model can help evaluate requirements that depend on meaning. In our citation example, the model can read the source passage and assess whether it supports the claim. This is often called using an LLM as a judge. The reviewer needs context just as the writer did: the task, the work being judged, the criteria, and the evidence relevant to those criteria.

A separate review call lets us select that context for the check. The reviewer need not receive every search attempt or the writer's explanation of why its draft is good. In the example below, the application supplies the actual claim and source passage, then asks for a judgment it can inspect. The callModel helper represents the same kind of model adapter used in earlier chapters; it formats these instructions and inputs into a request.

const review = await callModel({
  instructions: [
    "Does the source support the claim of current availability?",
    "Return pass, fail, or unknown.",
    "Include the claim ID, your explanation, and quoted evidence.",
    "If evidence is unavailable, return unknown.",
  ].join("\n"),
  input: {
    assignment: "Compare products available to use this week.",
    claim: {
      id: "claim-17",
      text: "Product B is available now.",
      sourceId: "source-4",
    },
    source: {
      id: "source-4",
      text: "Product B is scheduled for release next month.",
    },
  },
})

A useful finding identifies the problem and the evidence behind it. For this example, the reviewer should report that the source describes a future release and therefore does not support the claim of current availability. The application can represent that finding as a record. Parsing and validating a structured response checks its format; the quality of the judgment still depends on the review.

// An illustrative finding from the review above.
const finding = {
  criterion: "Source supports current availability",
  result: "fail",
  location: "claim-17",
  evidence: [
    "source-4: Product B is scheduled for release next month.",
  ],
  explanation: "The source describes a future release.",
}

Code checks and model reviews can both produce records in this form. Each record says what was checked, the result, and the supporting evidence. An unknown result preserves the difference between a failed requirement and a requirement we could not assess. A score can help compare drafts; a finding that identifies the faulty claim can help correct it. Choose the information in the assessment to support the decision that comes next.

Calibration

A reviewer can make mistakes too. A separate call gives the model a focused reviewing task, but it can still overlook the same ambiguity as the writer. It may reward confident prose or accept a citation without reading it closely. Self-review can help when it checks specific evidence, yet neither a fresh context nor an extra review guarantees a correct judgment.

Calibration means checking whether the evaluator's judgments match the standard we intended. We can have people review a small set of examples, compare their findings with the evaluator's, and examine disagreements. The reasons for those judgments help us decide what needs to change:

• An unsupported claim passes: check whether the evaluator received the relevant source and understood what to look for.
• Reviewers disagree about a public beta: clarify whether the task counts a beta as a released product.
• A human reviewer misses evidence: correct the reference judgment used to assess the evaluator.

Different checks can work together. A source lookup can catch a missing citation before a model reviews the claim's meaning. A person can examine uncertain findings and a sample of accepted work. Sampling accepted work matters because an evaluator's missed mistakes appear among its successes. Checking a test or validator follows the same principle: use examples whose expected results are known, including cases that should fail.

Reliability

Evaluating one result tells us about that attempt. To understand how reliably the system works, we need to run it on a range of tasks. Model responses can also vary when the task stays the same, so repeated attempts can reveal failures that one successful demonstration misses.

A small evaluation set can start with variations of the citation example:

• A release note clearly confirms that the product is available.
• An announcement describes a release planned for the future.
• Two sources disagree, or an older source has been superseded.
• The necessary source or publication date is missing.

Run the system on those cases and keep the assessments with each result. Repeating selected cases shows whether the same situation produces inconsistent answers. Grouping failures by cause shows where the system struggles: a high overall score can conceal repeated mistakes on future announcements. If time or cost matters to the task, record those measurements alongside the result so improvements in one dimension remain visible against the others.

Evaluation can help at several points:

• Before a change: run saved cases to compare a proposed prompt or tool with the current system.
• During a task: check citations in a draft before returning it.
• After use: review actual outcomes to find failures the original test set never covered.

The right amount of evaluation depends on the work. A quick lookup may be worth running for every citation, while a detailed model review takes more time and a human review requires someone's attention. We can combine inexpensive checks with selective review, then use what we learn from real failures to improve the cases and criteria.

Evaluation gives the system a judgment about its work and, when possible, evidence explaining that judgment. Saving the finding does not correct the draft. The next chapter, Feedback, connects those findings to the next model call or action so the system can respond to what it learned.