Forgetting to Remember

Memento is a film about a man who cannot make new memories. It is also the most useful fable I know for the way we build with language models.

Leonard Shelby navigates by Polaroids, tattoos, and scribbled notes. His method is simple: observe, summarize, tattoo, execute. The system is ingenious until a note is wrong. Then it becomes a machine for producing certainty without truth.¹The notes in Memento are not a neutral record. Leonard writes some of them after the fact, and the film deliberately withholds the source of key information. The system is vulnerable because its author is also a suspect.↩

We are building the same machine at scale.

The Finite Window

A million-token context is a bigger desk. The mind still has to decide what to keep on it.

The context window is sold as memory, but it is closer to a per-inference budget: the amount of text a model can attend to in one pass.²A context window is a limit on the tokens available to a single inference pass. A KV cache is a computational shortcut for attention over tokens that have already been processed. Neither is durable memory in the sense an agent needs.↩ When the conversation outgrows the window, something must be dropped, compressed, or retrieved. The model does not experience that loss. It continues speaking with the confidence of a system that has never seen the missing page.

Two failure modes follow. The first is lossy compaction. A summary preserves the shape of a conversation and loses its conditions: the exception, the caveat, the reason a decision was reversed. The second is poisoned retrieval. A retrieval-augmented generation (RAG) pipeline can pull a stale document, a near-duplicate, or a passage that is relevant in vocabulary and wrong in context. The model then reasons from corrupted state. Its failures arrive with total conviction.

A larger window postpones these problems. It does not remove them.

Meta-Conway’s Law

Conway’s Law observes that organizations tend to design systems that mirror their own communication structures. The architecture of the software ends up carrying the shape of the meetings, teams, and reporting lines that produced it.

The AI version is a useful provocation: the architecture of the work must conform to the token bounds of the models executing it.³Conway’s Law is a heuristic about organizations, not a law of nature. “Meta-Conway’s Law” is a provocation, not an established theorem.↩ A long task becomes a sequence of bounded jobs, each with a clean context, an explicit contract, and a place to hand off state.

This is why agent systems keep rediscovering the same furniture: task decomposition, checkpoints, serialized state, handoffs, and audit trails. The comparison to memory management is apt. Context is scarce, allocation matters, and leaks are expensive. An organization that ignores this will build systems that look elegant in a demo and fail on the third handoff.

The Autonomous Cliff

The popular picture of autonomous AI is a mind that runs indefinitely, improving itself in a feedback loop. The Memento problem interrupts that picture.

Long-horizon autonomy requires memory. Memory requires compression. Compression introduces drift. Drift compounds. A system can be brilliant for an hour and incoherent by Thursday. It does not need to become malevolent; it only needs to confuse its summary with the world.

This does not make advanced AI impossible. It makes one particular picture of it—the sealed, self-contained mind running forever without external scaffolding—much harder to sustain.⁴Agents can extend beyond a single context with files, tools, checkpoints, and human review. The argument concerns self-contained autonomy without such scaffolding, not the impossibility of long-horizon systems.↩ A system that must compress its own state is structurally vulnerable to drift. Let it run long enough, and it may act with absolute certainty on a wildly distorted version of its original goal.

It does not need to become malevolent. It only needs to confuse its summary with the world.

The Human as Orchestrator

The human’s role moves from generation to orchestration.

That means three things. Break the work into discrete, high-density units that fit inside a clean context. Audit the summaries and verify the retrieval against something outside the model. Feed clean input into each batch, with the provenance needed to know where it came from.

This is less glamorous than alignment research. It is closer to being a good editor. The orchestrator decides what enters the window, what survives summarization, and what must be checked against the world.

The human advantage is access to evidence outside the system’s own summary. Human oversight is not a guarantee—people are tired, captured, and wrong—but it is a structural check on a system that otherwise has no way to notice what it has forgotten.

Don’t Trust the Tattoos

Memento is a story about memory, and also about evidence. Leonard’s notes can be manipulated, and he manipulates himself. The same vulnerability scales with our systems: a summary can be poisoned, a retrieval can be gamed, a log can be incomplete.

Build verification into the loop: provenance, checkpoints, external truth, and the authority to say that a note is wrong.

The machine can hold the context. The human has to hold the standard.