---
title: "Evaluation | Elements of Agentic System Design"
description: "How to establish whether the work succeeded."
---

- [Idyllic Labs](https://idylliclabs.com/)

[Writing](https://idylliclabs.com/writing)

[Elements of Agentic System Design](https://idylliclabs.com/projects/elements-book)

Chapter 8, second edition working draft

# Evaluation

How to establish whether the work succeeded.

![A plumb line beside a stone arch makes the arch's alignment visible against a fixed reference.](https://idylliclabs.com/images/elements-book/chapter-evaluation.png)

Evaluation compares the work with a requirement using evidence that can be inspected.

Contents

[Introduction](https://idylliclabs.com/projects/elements-book)

- [1. Context](https://idylliclabs.com/projects/elements-book/context)
- [2. Memory](https://idylliclabs.com/projects/elements-book/memory)
- [3. Agency](https://idylliclabs.com/projects/elements-book/agency)
- [4. Reasoning](https://idylliclabs.com/projects/elements-book/reasoning)
- [5. Coordination](https://idylliclabs.com/projects/elements-book/coordination)
- [6. Artifacts](https://idylliclabs.com/projects/elements-book/artifacts)
- [7. Autonomy](https://idylliclabs.com/projects/elements-book/autonomy)
- [8. Evaluation](https://idylliclabs.com/projects/elements-book/evaluation)[Criteria](#evaluation-criteria)[Evidence](#artifacts-traces-and-evidence)[Tests and validators](#tests-and-validators)[Model-based review](#evaluation-methods-and-calibration)[Calibration](#calibration)[Reliability](#reliability-across-cases-and-trials)
- [9. Feedback](https://idylliclabs.com/projects/elements-book/feedback)
- [10. Learning](https://idylliclabs.com/projects/elements-book/learning)

Evaluation is how we judge whether an agent's work meets the task's requirements. The systems built so far can keep working across many model calls, but finishing a run does not establish that the task succeeded. A convincing answer may contain a mistake that the agent never noticed. To check the work, we need to decide what counts as success, obtain evidence about the result, and compare the two.

## Criteria

A criterion is a condition we check to judge whether the agent did its job. It connects the task we gave the agent to a result we can inspect:

- **A research agent's answer:** the cited sources support the factual claims in the answer.
- **A coding agent's bug fix:** a test that reproduces the reported bug passes against the edited code, and existing tests still pass.
- **An agent's document update:** the saved document contains the requested change.

A task may need several criteria. The research answer may need to cover every part of the question as well as support its claims. A set of criteria is often called a rubric. The rubric tells the evaluator what to check while allowing different answers or approaches to satisfy the task. Keeping the results for each criterion makes it clear where the work fell short, which an overall score alone cannot explain.

![A technology comparison assignment expands into criteria for coverage, evidence, and usefulness, each linked to an observation.](https://idylliclabs.com/images/elements-book/evaluation-02.png)

Quality becomes testable when requirements name what must be observed, rather than merely asking for a good answer.

## Evidence

Once we know what to check, we need the material that can answer the question. To assess a factual claim in the research answer, the evaluator needs the claim and its supporting source. The agent's assurance that it checked the source adds little if we cannot inspect what the source says. This connects evaluation to the artifacts from earlier chapters: saved work gives a reviewer something to examine after the run ends.

Different requirements call for different evidence:

- **The final answer:** the claims, explanations, and formatting the reader receives.
- **The resulting state:** the actual file, document, or database record after the agent's action.
- **The execution trace:** the recorded operations, their results, and the time or resources they consumed.

![An assistant claims to have updated and published a briefing. A trace shows publish not called, the saved file shows revision 13, and the website revision is unknown. The assessment is updated, publication unconfirmed.](https://idylliclabs.com/images/elements-book/evaluation-10-v2.png)

The saved briefing confirms an edit. The trace records no publish call, and the destination has not been checked, so publication remains unconfirmed.

If an agent says it fixed a bug, we can run tests against the changed files and record the results. Asking the model whether those tests would pass gives us its prediction. Running them gives us observations about the code.

The trace helps when the requirement concerns how the work was done. If a comparison must examine both company filings and product documentation, the recorded results can show whether the agent received relevant passages from both collections. A search call alone does not establish that coverage: we need to inspect what it returned. Several orders of searching and reading can satisfy the requirement, so the check should allow those alternatives unless the order itself matters.

![A test run on edited files produces exit status zero and twelve passing assertions. Below, two different research orders both return relevant passages from collections A and B, while an A-only route returns no B passage and leaves coverage missing.](https://idylliclabs.com/images/elements-book/evaluation-11-v2.png)

Running tests produces observations about the edited files. Research coverage likewise depends on the relevant passages returned, even when the operations occur in a different order.

## Tests and validators

When a requirement has a precise rule, ordinary code can check it. A validator is a function that checks whether data follows such a rule. For citations, we can look up each source identifier in the stored sources. The example below receives a claim with a source identifier and a map of available sources. Its result names the condition checked and the claim it applies to.

```typescript
function checkCitation(claim, sources) {
  const exists = sources.has(claim.sourceId)

  return {
    criterion: "Citation points to a stored source",
    result: exists ? "pass" : "fail",
    location: claim.id,
    evidence: exists ? [claim.sourceId] : [],
  }
}
```

This check can pass while the claim is wrong. A research answer written on July 1 might say “Product B is available now,” while a source published that day says “Product B is scheduled for release next month.” The source describes a planned August release, so it does not support availability on July 1. The lookup correctly establishes that the source exists. Checking whether it supports the claim requires reading what the passage means. A check can work exactly as intended while leaving an important requirement untested.

![The identifier on claim 17 resolves to source 4, while a separate question asks whether source 4 supports the claim.](https://idylliclabs.com/images/elements-book/evaluation-03.png)

A citation lookup can establish that a source exists. Source support requires a further check.

## Model-based review

A language model can help evaluate requirements that depend on meaning. In our citation example, the model can read the source passage and assess whether it supports the claim. This is often called using an LLM as a judge. The reviewer needs context just as the writer did: the task, the work being judged, the criteria, and the evidence relevant to those criteria.

A separate review call lets us select that context for the check. The reviewer need not receive every search attempt or the writer's explanation of why its draft is good. In the example below, the application supplies the actual claim and source passage, then asks for a judgment it can inspect. The `callModel` helper represents the same kind of model adapter used in earlier chapters; it formats these instructions and inputs into a request.

```typescript
const review = await callModel({
  instructions: [
    "Does the source support the claim of current availability?",
    "Return pass, fail, or unknown.",
    "Include the claim ID, your explanation, and quoted evidence.",
    "If evidence is unavailable, return unknown.",
  ].join("\n"),
  input: {
    assignment: "Compare products available to use this week.",
    assessmentDate: "2026-07-01",
    claim: {
      id: "claim-17",
      text: "Product B is available now.",
      sourceId: "source-4",
    },
    source: {
      id: "source-4",
      publishedAt: "2026-07-01",
      text: "Product B is scheduled for release next month.",
    },
  },
})
```

![Generation produces a draft; a separate reviewing call receives the draft, assignment, rubric, and sources and returns findings with supporting passages.](https://idylliclabs.com/images/elements-book/evaluation-06.png)

A fresh reviewing context can focus on the assignment and evidence. It still needs calibration and can share the generator’s errors.

A useful finding identifies the problem and the evidence behind it. For this example, the reviewer should report that the source describes a future release and therefore does not support the claim of current availability. The application can represent that finding as a record. Parsing and validating a structured response checks its format; the quality of the judgment still depends on the review.

```typescript
// An illustrative finding from the review above.
const finding = {
  criterion: "Source supports current availability",
  result: "fail",
  location: "claim-17",
  evidence: [
    "source-4: Product B is scheduled for release next month.",
  ],
  explanation: "The source describes a future release.",
}
```

Code checks and model reviews can both produce records in this form. Each record says what was checked, the result, and the supporting evidence. An `unknown` result preserves the difference between a failed requirement and a requirement we could not assess. A score can help compare drafts; a finding that identifies the faulty claim can help correct it. The information an assessment retains depends on the decision it will support.

![Assignment, artifact, trace, sources, and rubric enter an evaluator that returns named checks with pass, fail, or unknown results and supporting evidence.](https://idylliclabs.com/images/elements-book/evaluation-01.png)

An evaluator combines observations with criteria and returns findings. A controller decides how to use them.

## Calibration

A reviewer can make mistakes too. A separate call gives the model a focused reviewing task, but it can still overlook the same ambiguity as the writer. It may reward confident prose or accept a citation without reading it closely. Self-review can help when it checks specific evidence, yet neither a fresh context nor an extra review guarantees a correct judgment.

Calibration means checking whether the evaluator's judgments match the standard we intended. We can have people review a small set of examples, compare their findings with the evaluator's, and examine disagreements. The reasons for those judgments help us decide what needs to change:

- **An unsupported claim passes:** check whether the evaluator received the relevant source and understood what to look for.
- **Reviewers disagree about a public beta:** clarify whether the task counts a beta as a released product.
- **A human reviewer misses evidence:** correct the reference judgment used to assess the evaluator.

Different checks can work together. A source lookup can catch a missing citation before a model reviews the claim's meaning. A person can examine uncertain findings and a sample of accepted work. Sampling accepted work matters because an evaluator's missed mistakes appear among its successes. Checking a test or validator follows the same principle: use examples whose expected results are known, including cases that should fail.

![Uncertain findings and sampled passes reach human review. Disagreement leads to an investigation of the same work and evidence, which can reveal an evaluator error, an unclear criterion, or a human review error.](https://idylliclabs.com/images/elements-book/evaluation-07-v2.png)

Review uncertain findings and a sample of apparent successes. When judgments disagree, inspect the evidence before deciding which judgment or criterion needs correction.

## Reliability

Evaluating one result tells us about that attempt. To understand how reliably the system works, we need to run it on a range of tasks. Model responses can also vary when the task stays the same, so repeated attempts can reveal failures that one successful demonstration misses.

A small evaluation set can start with variations of the citation example:

- A release note clearly confirms that the product is available.
- An announcement describes a release planned for the future.
- Two sources disagree, or an older source has been superseded.
- The necessary source or publication date is missing.

Running the system on these cases produces a set of results and assessments we can compare. Repeating selected cases shows whether the same situation produces inconsistent answers. Grouping failures by cause shows where the system struggles: a high overall score can conceal repeated mistakes on future announcements. Timing and cost measurements help establish whether an improvement in quality is worth the additional work.

![An illustrative task-by-trial matrix separates routine, conflicting-source, missing-date, and insufficient-evidence cases, with different outcomes visible by category.](https://idylliclabs.com/images/elements-book/evaluation-08.png)

Repeated trials and failure categories reveal weaknesses that an average score can hide.

Evaluation can help at several points:

- **Before a change:** run saved cases to compare a proposed prompt or tool with the current system.
- **During a task:** check citations in a draft before returning it.
- **After use:** review actual outcomes to find failures the original test set never covered.

The right amount of evaluation depends on the work. A quick lookup may be worth running for every citation, while a detailed model review takes more time and a human review requires someone's attention. We can combine inexpensive checks with selective review, then use what we learn from real failures to improve the cases and criteria.

![Pre-deployment tests, checks before publication, and production review operate at different moments and feed decisions.](https://idylliclabs.com/images/elements-book/evaluation-09.png)

Before release, during a run, and after use are different opportunities to inspect and improve the system.

Evaluation gives the system a judgment about its work and, when possible, evidence explaining that judgment. Saving the finding does not correct the draft. The next chapter, Feedback, connects those findings to the next model call or action so the system can respond to what it learned.

[Previous Autonomy](https://idylliclabs.com/projects/elements-book/autonomy)

[Next Feedback](https://idylliclabs.com/projects/elements-book/feedback)

[Idyllic Labs](https://idylliclabs.com/)

[Writing](https://idylliclabs.com/writing)

[Markdown](https://idylliclabs.com/projects/elements-book/evaluation/index.md)