---
title: "Memory | Elements of Agentic System Design"
description: "How information survives and becomes available again."
---

- [Idyllic Labs](https://idylliclabs.com/)

[Writing](https://idylliclabs.com/writing)

[Elements of Agentic System Design](https://idylliclabs.com/projects/elements-book)

Chapter 2, second edition working draft

# Memory

How information survives and becomes available again.

![An open stone archive contains many recessed shelves, with a few selected tablets laid on a reading table.](https://idylliclabs.com/images/elements-book/chapter-memory.png)

Stored records outlast a call. Retrieval brings selected records back into context.

Contents

[Introduction](https://idylliclabs.com/projects/elements-book)

- [1. Context](https://idylliclabs.com/projects/elements-book/context)
- [2. Memory](https://idylliclabs.com/projects/elements-book/memory)[Three design questions](#three-design-questions)[Conversation history](#conversation-history)[Decision records](#decision-records)[User profiles](#user-profiles-and-preferences)[Folders and text search](#document-hierarchies-and-text-search)[Skills for procedural memory](#skills-for-procedural-memory)[Journals](#journals-of-observations-and-decisions)[Semantic search](#vector-databases-and-semantic-search)[Retrieval quality](#retrieval-quality-and-interpretation)
- [3. Agency](https://idylliclabs.com/projects/elements-book/agency)
- [4. Reasoning](https://idylliclabs.com/projects/elements-book/reasoning)
- [5. Coordination](https://idylliclabs.com/projects/elements-book/coordination)
- [6. Artifacts](https://idylliclabs.com/projects/elements-book/artifacts)
- [7. Autonomy](https://idylliclabs.com/projects/elements-book/autonomy)
- [8. Evaluation](https://idylliclabs.com/projects/elements-book/evaluation)
- [9. Feedback](https://idylliclabs.com/projects/elements-book/feedback)
- [10. Learning](https://idylliclabs.com/projects/elements-book/learning)

Memory preserves information beyond the context of a single model call. A conversation, project, or collection of sources can grow larger than one context window. The system therefore needs somewhere to keep that information and a way to bring the useful parts back into context. What the model appears to remember depends on what the system saves, what it retrieves, and how the next call interprets it.

## Three design questions

- **What should be stored?** The record needs to become useful input for a later model call. In the text examples here, that means retaining text or data that can be presented as text. Notes, observations, and explanations of failed attempts can all be useful if they preserve enough information for the model to interpret them in a new situation.
- **How should it be stored?** How the information will be retrieved helps determine its format. A Markdown file suits reading a profile or procedure; database rows make filtering events by user and time straightforward. The storage format need not match the input format: selected fields can become a passage in context. The useful choice is the one that makes the needed information easy to recover.
- **How should the model find and access it?** Folder names, short descriptions, and searchable indexes help the model recognize relevant material before reading it. The application can supply a small catalog or frequently needed facts at the start, then retrieve other material when the model requests it. Preloading saves discovery steps but consumes context; access on demand saves space but requires a way to find what is missing.

![A source record with identity, publication date and text is serialized into the evidence section of a model input while the original remains in storage.](https://idylliclabs.com/images/elements-book/memory-06.png)

A retrieved record becomes text in the next context. The model interprets the passage alongside its source, date, and the current question.

## Conversation history

Conversation history is the most familiar example. An application saves each message with its role, such as user or assistant. When a new request arrives, it supplies the earlier messages along with that request. The model reads the exchange and responds as its continuation. For a short conversation, the saved history can be an array that the application persists between requests.

```typescript
const history = await conversations.load(taskId)
const messages = [
  ...history,
  { role: "user", content: "Add the new filing." },
]
const response = await callModel({ messages })
await conversations.save(taskId, [
  ...messages,
  { role: "assistant", content: response.text },
])
```

Longer histories can live in an event log, where each entry records a message or tool result with its task and time. Code can query recent turns instead of loading the entire log. This is already a retrieval policy: recency decides what the next call sees. Selection must preserve usable exchanges, including the results associated with tool calls. Earlier events remain in storage even when they no longer fit in the current input.

![Four timestamped conversation events remain stored. The last two are selected into a messages array alongside a new request. Both the array and a separate retained task reach the next model call.](https://idylliclabs.com/images/elements-book/memory-09.png)

Stored events preserve the exchange. Selected turns keep their roles and order, while a retained task record carries the continuing assignment.

## Decision records

Recent history eventually leaves important decisions out. A separate note can retain a decision and its reason after the conversation moves on. “Used the cached filing” records an action. “The live endpoint returned a rate-limit error; used the cached July filing” also preserves why the workaround was chosen. A later call can interpret that reason and judge whether to try the live endpoint again. The note needs enough evidence to support that judgment, even if its wording is rough.

## User profiles

A user profile can begin as a Markdown note. If a user asks for linked sources throughout a project, the application can save that preference in `users/U7/profile.md` and read it when preparing later requests. The note can preserve the user's wording, the project it applies to, and the message that established it. A model can interpret those details without a separate database field for every possible preference.

```markdown
# User U7

## Project P2
Use linked sources in research briefings.
Requested in message M8 on 2026-07-01.

## Report B-17
Use footnotes for this report.
Requested in message M19; other reports keep linked sources.
```

The note records where each preference applies and where it came from. The report-specific request overrides the project preference for one report; it does not establish a new preference for everything. Retaining the source message lets the application check or correct the note. As the profile grows, it can split into sections or files so a request about one project does not load every fact about the user.

![An explicit user instruction becomes a project-scoped preference record. A later report-specific instruction overrides it for one report while other reports retain the project preference.](https://idylliclabs.com/images/elements-book/memory-05.png)

A stored preference should preserve who supplied it, where it applies, and the evidence for changing it.

## Folders and text search

Meaningful paths provide a simple index. A user identifier locates a profile; a project directory collects decisions and open questions; a dated filename locates a journal entry. A directory listing tells the model what records exist without supplying their full contents. The application can read a known path directly, or let the model choose a file to inspect through a reading tool.

```text
memory/
  users/U7/profile.md
  projects/P2/decisions.md
  projects/P2/open-questions.md
  journal/2026-07-01.md
  journal/2026-07-02.md
```

When the path is unknown, a text search such as `grep` or `rg` can find matching lines across the files. Paths and line numbers let the model request the surrounding passage when a match is too short to interpret. This gives a small collection a practical retrieval process: inspect names, search content, then read the useful part. A filesystem can support selective retrieval before a specialized search index is needed.

![A folder tree contains a user profile, project decisions, and a dated journal. A known-path read selects the profile and a text search selects a journal line; both reach context while other files remain on disk.](https://idylliclabs.com/images/elements-book/memory-07.png)

A known path locates a profile; text search locates a journal passage. Only the selected text enters the next context.

## Skills for procedural memory

Some saved knowledge describes how to work. A skill can store a procedure in a `SKILL.md` file, with supporting scripts or examples nearby. A system with many skills cannot afford to include every full procedure in every context. It can instead supply a small catalog of names, descriptions, and locations. The model uses that metadata to judge which procedure is relevant to the current task.

```typescript
const skillCatalog = [
  {
    name: "compare-filings",
    description: "Compare company filings across reporting periods",
    path: "skills/compare-filings/SKILL.md",
  },
  {
    name: "review-citations",
    description: "Check whether source passages support a draft's claims",
    path: "skills/review-citations/SKILL.md",
  },
]
```

For a filing comparison, the model can request the first file. Code reads it, and the next call receives the full instructions while the other skill remains only a catalog entry. The description matters because the model must recognize a useful procedure before reading it. This separates discovery from loading: a small index helps choose which larger body of knowledge should enter context. Agency will explain how a model's request becomes the file read; loading a procedure does not itself execute its steps.

![A catalog contains compare-filings and review-citations. The comparison task selects compare-filings, a file read loads its SKILL.md, and its instructions enter the next context. The unused skill stays a catalog entry.](https://idylliclabs.com/images/elements-book/memory-08.png)

The model sees skill names and descriptions, selects a relevant procedure, and receives its full instructions through a file read.

## Journals

Profiles and procedures collect knowledge expected to remain useful. A journal instead preserves observations and decisions as work unfolds. Each entry can record its time, project, what happened, and links to evidence. Unlike the raw conversation log, it selects information worth keeping for future work. “What did we decide yesterday?” can use a date range; a question about a known endpoint can use text search.

## Semantic search

Text search can miss an entry when the question and the entry use different words. “Why did the analysis get cheaper?” might need a note that says “Switched to discounted batch requests after the rate-limit errors.” Semantic search offers another retrieval route. An embedding model converts entries and queries into vectors that an index compares for similarity. The index returns candidate passages, so a related idea can be found even when the query shares few exact words with the note.

A search can find more candidates than the next context should contain. Reranking compares the candidates more closely with the current question, often using a model trained or prompted to assess relevance. Code can then choose a small set within the context budget. A recent entry, a semantically related older entry, and a standing preference can all matter for different reasons. The selected passages and their sources enter context for the model to interpret together. Retrieving material to support generation is commonly called retrieval-augmented generation, or RAG.

![A briefing question produces recent conversation, similar source passages, and a durable citation instruction. The selected set retains examples from the useful routes while leaving other candidates out.](https://idylliclabs.com/images/elements-book/memory-02.png)

Similarity, recency, and task importance can propose different records. The selection policy determines which reach context.

## Retrieval quality

Relevance is a judgment about usefulness to a question, not proof that a passage is true or current. An old filing can closely match a question about this quarter's revenue. Filtering by company, reporting period, and access rights narrows the eligible material; retaining those fields beside the text lets the next call understand its scope. A successful search can still produce a wrong answer if those distinctions disappear when the passage enters context.

```typescript
const evidenceText = selectedRecords.map(record => {
  const source = "Source: " + record.sourceId
  const period = "Reporting period: " + record.reportingPeriod
  const date = "Published: " + record.publishedAt
  return [source, period, date, record.text].join("\n")
}).join("\n\n")

// Text supplied to the next model call:
// Source: S3
// Reporting period: 2026 Q2
// Published: 2026-07-01
// Operating costs fell.
```

The model also has to reconcile what the retrieved passages say. A journal entry may record a temporary workaround, while a newer source shows that the original limitation has been removed. The model can use both to reconsider the earlier decision. It can also misread a date, miss a negation, or treat an inference as a fact.

![A schematic embedding space places an old and a current filing near a revenue query. A metadata filter excludes the old filing and retains the current one.](https://idylliclabs.com/images/elements-book/memory-03.png)

A nearby embedding suggests a candidate. Date and project fields can still exclude it from the current task.

Different questions therefore benefit from different routes through the same stored material. A known profile needs a path lookup, a recent exchange needs a history query, and an unfamiliar description may need semantic search followed by reranking. The model can help choose the route and judge the results. The application still has to perform the reads and supply the selected text in a later call.

Summaries save space by retaining an interpretation of earlier material. Keeping references to the original lets a later call recover details that a new question makes relevant. The same principle applies to profiles and journals: preserve the reasons, dates, and scope that allow future interpretation, then provide retrieval paths that can find them. The storage choice follows what those future calls need to read.

![A dated note records that Briefing B-17 used a cached July filing because the live endpoint was rate-limited. A shorter summary keeps the note reference D7. A later question follows D7 to recover the old reason before checking the live endpoint again.](https://idylliclabs.com/images/elements-book/memory-04-v2.png)

A summary can retain a reference to the original decision note. Reading that note recovers why the decision was made; current conditions still require a fresh check.

We now have ways to preserve a conversation, retain knowledge, and retrieve useful material without loading it all at once. The model can identify a missing source or request a procedure, but its response does not perform the read. The next chapter connects those decisions to executable operations so the model can choose what information to obtain and what actions to take.

[Previous Context](https://idylliclabs.com/projects/elements-book/context)

[Next Agency](https://idylliclabs.com/projects/elements-book/agency)

[Idyllic Labs](https://idylliclabs.com/)

[Writing](https://idylliclabs.com/writing)

[Markdown](https://idylliclabs.com/projects/elements-book/memory/index.md)