Designing Agents Around the KV Cache

[!NOTE] TL;DR

An agent re-sends its whole context on every step. Input tokens dwarf output tokens, so prefix caching decides your cost and latency.

Caches match on an exact prefix. One changed token near the top, like a timestamp, a random ID or a reordered tool list, throws away everything after it.

The rules follow from that: a stable prefix, append-only history, deterministic serialisation, a fixed tool list that you mask instead of editing, and volatile data at the end.

Compact at checkpoints into a new prefix that then stays stable, not on every turn. Move big observations into files and keep handles.

Measure it. Every provider returns cached-token counts in the usage block, so track the hit rate per run and treat a drop as a bug.

Most agent cost discussions are about the model: which one, how big, how many reasoning tokens. That matters less than people think. In a long-running agent, the thing you pay for most is the model re-reading the same context, step after step.

This post covers how I design harnesses around that fact. I start with what the KV cache actually is, then how inference servers and API providers reuse it, then the engineering rules that follow. Most of those rules are boring once you see the mechanism. They also get broken constantly.

Why agents are an input-token problem

A chat turn is short: a prompt goes in, an answer comes out. An agent loop is different. Every step sends the full history (tools, system prompt, task, every action and every observation so far), and the model replies with one small tool call.

Here is some illustrative arithmetic, not a measurement. Take a 50-step run with a 10k-token prefix (tools plus system prompt), 1k tokens of new observation per step and about 200 output tokens per step.

Input across the run: 50 × 10k + 1k × (0 + 1 + … + 49) ≈ 500k + 1.2M ≈ 1.7M tokens.
Output across the run: 50 × 200 = 10k tokens.

That is a ratio of about 170:1. Output pricing barely matters at that ratio. What matters is whether those 1.7M input tokens get computed from scratch or come out of a cache.

The input bar grows every step because the whole history is re-sent. The output bar stays tiny. Numbers are illustrative.

What the KV cache actually is

A transformer layer's attention computes a query, a key and a value vector for every token. To produce the next token, the model attends from the new token's query to the keys and values of every earlier token, in every layer. Those K and V tensors are the KV cache.

Generation has two phases:

Prefill processes the whole prompt in one parallel pass and produces K/V for every prompt token. It is compute-bound, and its cost grows with prompt length.
Decode generates one token at a time. It reuses the stored K/V and only computes K/V for the new token. It is memory-bandwidth-bound.

The key property is that a token's K/V depends on that token and every token before it, and on nothing after it. So if two requests share an identical prefix, the K/V for that prefix is identical too, and you can reuse it without recomputing. If the prefix differs by a single token at position i, everything from i onwards has to be recomputed.

That is the whole game: reuse is by exact prefix.

Prefix caching in inference servers

If you host your own models, the server does the reuse for you, as long as you send it reusable prefixes.

vLLM has automatic prefix caching. It splits the KV cache into fixed-size blocks and identifies each full block by a hash of its tokens together with everything before it. A new request that starts with the same tokens finds those blocks already resident and skips their prefill. Recent versions turn it on by default. In older ones you enable it with a flag.

SGLang introduced RadixAttention. It keeps cached KV in a radix tree keyed by token sequences and evicts least-recently-used leaves. Shared prefixes become shared nodes, so a fan-out of requests that share a system prompt, or many branches of one agent run, all hang off the same cached trunk.

A radix tree of cached prefixes. Two branches of task A share everything up to step 8. Every request shares the tools and system trunk.

One operational detail matters when you run more than one replica: the cache is local to a replica. If step 7 of a run lands on a different worker than step 6, you pay full prefill. Route requests by session ID so that a run sticks to one worker.

Provider prompt caching

Hosted APIs expose the same mechanism with different ergonomics.

Anthropic makes prompt caching explicit. You mark breakpoints with cache_control: {"type": "ephemeral"} on content blocks, and everything up to and including a marked block becomes a cacheable prefix. The prefix order is tools, then system, then messages. You get a small number of breakpoints per request. The default lifetime is short, a few minutes, and each hit refreshes it. A longer-lived option costs more to write. Cache writes cost somewhat more than normal input, and cache reads cost a small fraction of it.

OpenAI caches automatically for prompts above a minimum length, with no markup. You can't place breakpoints, so the ordering discipline in this post is the only lever you have. Cached input tokens are billed at a discount.

Both also have a minimum cacheable length (on the order of a thousand tokens, depending on the model), so tiny prompts never hit. That's rarely a problem for agents.

[!TIP] Prices and TTLs change. Read the current numbers from the provider's docs rather than from blog posts, this one included. The shape stays the same: reads are much cheaper than uncached input, and writes cost a little extra.

Credit where it's due

Much of this became common knowledge through the Manus team's 2025 post, Context Engineering for AI Agents: Lessons from Building Manus. It argued that KV-cache hit rate is the single most important metric for a production agent. It also recommended masking tools instead of removing them and using the file system as externalised context. I agree with all three. The rest of this post is my version of how to enforce them in code.

The rules

Each rule below follows directly from "reuse is by exact prefix".

Keep the prefix byte-stable

The tools block and the system prompt are the most-reused tokens in the run, because every step and every sibling run shares them. Treat them as a build artifact. Don't template them per request.

The common offenders:

A timestamp at the top of the system prompt, like Current time: 2026-03-11T10:42:07Z. Every second is a new prefix.
Request IDs, trace IDs or random nonces in the system prompt.
A tool list built from a set or a dict that isn't ordered, so the order changes between processes.
A user profile or feature flags rendered into the system prompt that can change in the middle of a session.

The same content in two layouts. With the clock at the end, only the last block is new. With the clock on top, the whole request is a miss on every step.

If the model needs the time, put it in the newest message. Cut the precision to what the task needs, too: a date changes once a day, a timestamp to the second changes on every call.

Make the context append-only

Never rewrite earlier turns. That includes "harmless" edits: trimming an old tool result, redacting a field after the fact, re-rendering history with a newer template, or dropping a failed step to keep things tidy.

Failed steps are worth keeping for another reason. Leaving the error in the history shows the model what didn't work, so it's less likely to try the same thing again.

Serialise deterministically

Anything you turn into text has to come out byte-identical every time. That covers tool results, observations and structured state. For JSON that means sort_keys=True, fixed separators and no floats that print differently across code paths. Python dicts keep insertion order, but insertion order often depends on the order an upstream API returned fields, and you don't control that.

Don't add or remove tools mid-run. Mask them.

Tool definitions sit at the very front of the prefix. Changing the tool list, even by loading a "relevant" subset per step, invalidates everything. It also leaves earlier turns referring to tools that no longer exist, which confuses the model.

Define the full tool set once per agent and keep it fixed. Restrict what is allowed per step another way:

Self-hosted: constrain decoding. Mask the logits of disallowed tool names, or prefill the start of the assistant turn so it can only continue into the allowed tools. Consistent name prefixes such as browser_* and shell_* make group masks easy.
Hosted APIs: use the provider's tool-choice controls where they fit, or state the allowed set in the newest message, and enforce it in the harness. A disallowed call gets a tool error back. It doesn't get executed.

Put volatile things at the end

Order the context by how often each part changes, slowest first: tools, system prompt, task, history, then the current step's ephemeral data (time, budget left, allowed tools, scratch reminders). Volatile data at the tail only invalidates itself.

A context builder that enforces it

Conventions erode. I'd rather have the harness fail loudly. This builder keeps a fixed tools and system prefix and append-only history. It asserts that what it sent last time is still a byte-identical prefix of what it sends now.

import hashlib
import json
from typing import Any
 
 
def canonical(obj: Any) -> str:
    return json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
 
 
class PrefixChangedError(RuntimeError):
    pass
 
 
class ContextBuilder:
    def __init__(self, system: str, tools: list[dict]) -> None:
        self._system = system
        self._tools = sorted(tools, key=lambda t: t["name"])
        self._messages: list[dict] = []
        self._sent: tuple[int, str] | None = None
 
    def append(self, role: str, content: str | list[dict]) -> None:
        self._messages.append({"role": role, "content": content})
 
    def _digest(self, n_messages: int) -> str:
        h = hashlib.sha256()
        h.update(canonical(self._tools).encode())
        h.update(canonical(self._system).encode())
        for message in self._messages[:n_messages]:
            h.update(canonical(message).encode())
        return h.hexdigest()
 
    def build(self, tail: str | None = None) -> dict:
        if self._sent is not None:
            n_messages, digest = self._sent
            if self._digest(n_messages) != digest:
                raise PrefixChangedError("earlier context was mutated; the cache will miss")
        self._sent = (len(self._messages), self._digest(len(self._messages)))
        messages = [dict(m) for m in self._messages]
        if tail is not None:
            messages.append({"role": "user", "content": tail})
        return {"system": self._system, "tools": self._tools, "messages": messages}
 
 
def observation(result: dict) -> str:
    return canonical(result)

The tail argument is where volatile data goes: the current time, remaining budget, allowed tools for this step. It goes on the request but not into history, so the next step's prefix doesn't depend on it. Note that providers expect user and assistant turns to alternate, so in a real loop you fold the tail into the latest user or tool-result turn. The point is the same: it's the last thing in the request and it never becomes part of the stored prefix.

Here's how it behaves when someone "cleans up" an old observation:

ctx = ContextBuilder(system="You are a careful research agent.", tools=[{"name": "search"}])
ctx.append("user", "Find the release date of X.")
ctx.build()
 
ctx.append("assistant", "Searching.")
ctx.append("user", observation({"hits": 3, "top": "X ships in May"}))
ctx.build()
 
ctx._messages[2]["content"] = "[truncated]"
ctx.build()  # raises PrefixChangedError

In production I log the error and emit a metric instead of crashing. In tests it fails the build. Either way, nobody finds out three weeks later from the invoice.

Why hash instead of keeping the last payload?

Keeping the previous serialised payload and calling startswith works too, but it holds a second copy of a context that can be hundreds of kilobytes, on every step of every concurrent run. A digest over the first n messages is constant-size. The check is O(context) either way, and that's trivial next to a network round trip.

An Anthropic request with cache breakpoints

With explicit caching, I place three breakpoints: the last tool (caches the whole tools block), the system prompt, and the newest message (caches the history so far, so the next step reads it).

import anthropic
 
client = anthropic.Anthropic()
 
TOOLS = [
    {
        "name": "read_file",
        "description": "Read a file from the workspace.",
        "input_schema": {
            "type": "object",
            "properties": {"path": {"type": "string"}},
            "required": ["path"],
        },
    },
    {
        "name": "search",
        "description": "Search the web.",
        "input_schema": {
            "type": "object",
            "properties": {"query": {"type": "string"}},
            "required": ["query"],
        },
        "cache_control": {"type": "ephemeral"},
    },
]
 
SYSTEM = [
    {
        "type": "text",
        "text": "You are a careful research agent. Prefer primary sources.",
        "cache_control": {"type": "ephemeral"},
    }
]
 
response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    tools=TOOLS,
    system=SYSTEM,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Find the release date of X.",
                    "cache_control": {"type": "ephemeral"},
                }
            ],
        }
    ],
)
 
print(response.usage.cache_creation_input_tokens, response.usage.cache_read_input_tokens)

The breakpoint on the newest message moves forward each step. That's fine: the previous step's breakpoint wrote the history up to that point, and the new request starts with exactly those bytes, so it reads them back and only writes the new tail.

With an append-only context, step N's request starts with step N-1's request byte for byte. The cached share keeps rising as the run gets longer.

Measuring the hit rate

You can't manage what you don't graph. Both major APIs return cache counters in the usage block, but they count differently, and that trips people up:

Provider	Cache read	Cache write	Uncached input
Anthropic	`usage.cache_read_input_tokens`	`usage.cache_creation_input_tokens`	`usage.input_tokens` (excludes both cache fields)
OpenAI (Chat Completions)	`usage.prompt_tokens_details.cached_tokens`	not reported separately	`usage.prompt_tokens` minus cached (prompt tokens include cached)

from dataclasses import dataclass
 
 
@dataclass
class CacheStats:
    read: int = 0
    written: int = 0
    uncached: int = 0
 
    def record_anthropic(self, usage) -> None:
        self.read += usage.cache_read_input_tokens or 0
        self.written += usage.cache_creation_input_tokens or 0
        self.uncached += usage.input_tokens
 
    def record_openai(self, usage) -> None:
        details = usage.prompt_tokens_details
        cached = (details.cached_tokens or 0) if details else 0
        self.read += cached
        self.uncached += usage.prompt_tokens - cached
 
    @property
    def total_input(self) -> int:
        return self.read + self.written + self.uncached
 
    @property
    def hit_rate(self) -> float:
        return self.read / self.total_input if self.total_input else 0.0

I keep one of these per run and emit hit_rate when the run finishes, tagged with the agent and the prompt version. Two habits make it useful:

Alert on drops, not on absolute values. A healthy long run should sit high. A sudden fall after a deploy almost always means someone put something volatile into the prefix.
Look at step 2. On the first step the cache is cold by definition. If step 2 doesn't read back what step 1 wrote, the prefix isn't stable, and you can stop looking anywhere else.

[!WARNING] Short TTLs bite agents that wait. If a run pauses for human approval or a slow job longer than the cache lifetime, the next step pays full prefill plus a fresh write. Either accept that, use a longer TTL where it pays off, or send a cheap keep-warm request while you wait.

Compaction without wrecking the cache

Contexts eventually get too long, and you have to shrink them. The naive approach, summarising the oldest turns on every step, is the worst option for caching. It rewrites the front of the history on every call, so the hit rate stays near zero for the rest of the run.

Two strategies keep the cache intact.

Compact at checkpoints, not continuously. When the context crosses a threshold, build a new prefix: the same tools, the same system prompt, then a single summary message of the run so far, then continue appending as normal. You pay one cache write at the checkpoint, and every step after that hits again. Choose a threshold well below the window so it happens rarely.

Offload observations to files and keep handles. A 40k-token web page or a large query result doesn't need to live in context. Write it to the agent's workspace and put a short, stable reference in the history:

saved 41,203 tokens to /work/obs/0007-search.md (title: "X release notes", 312 lines)

The model can call read_file when it actually needs the content. Make that decision when the observation is first added, not later. Truncating an observation after it's in history is a rewrite. Writing a handle at append time is free. Because the file stays on disk, nothing is lost, and the compression can be reversed.

Sub-agents have their own caches

A sub-agent is a separate context, which makes it a separate cache. That works in your favour:

Give sibling sub-agents the same tools and system prefix, so a parallel fan-out shares one cached trunk (the radix tree above, in practice).
Put the per-child task after that shared prefix.
Return a compact result to the parent and append it as a normal observation. The parent's prefix stays untouched, and the child's long exploration never enters it.

Anti-patterns, before and after

Clock at the top.

# before
system = f"Current time: {datetime.now().isoformat()}\nYou are a research agent."
 
# after
system = "You are a research agent."
request = ctx.build(tail=f"Today is {date.today().isoformat()}.")

Order that depends on the environment.

# before
tools = [registry[name] for name in enabled_tool_names]  # a set
obs = json.dumps(api_response)
 
# after
tools = sorted((registry[n] for n in enabled_tool_names), key=lambda t: t["name"])
obs = canonical(api_response)

Swapping the tool list per step.

# before
request["tools"] = [t for t in ALL_TOOLS if t["name"] in allowed_now]
 
# after
request = ctx.build(tail=f"Allowed tools this step: {', '.join(sorted(allowed_now))}.")
if call.name not in allowed_now:
    ctx.append("user", tool_error(call, "tool not allowed in this phase"))

Trimming history in place.

# before
for m in ctx.messages[:-10]:
    m["content"] = m["content"][:500]
 
# after
path = workspace.write(f"obs/{step:04d}.md", raw)
ctx.append("user", f"saved {count_tokens(raw)} tokens to {path}")

In these snippets tool_error, workspace and count_tokens stand in for whatever your harness already has. The point is where the change happens, not the helpers.

Checklist

The tools block is fixed per agent and sorted by name. Nothing per-request is templated into it.
The system prompt has no timestamps, IDs, nonces or per-user data that can change mid-session.
History is append-only, and the harness asserts it with a prefix hash.
Everything turned into text uses deterministic serialisation (sort_keys, fixed separators).
Tool availability is masked (constrained decoding, or the allowed set in the tail plus enforcement in the harness), never edited.
Volatile data (time, budget, allowed tools) goes in the tail and is not stored in history.
Large observations go to files at append time, and only a handle goes in context.
Compaction happens at checkpoints and produces a new prefix that then stays stable.
Sibling sub-agents share one prefix, and their results come back as compact observations.
Self-hosted: prefix caching is on and requests are routed to replicas by session.
Anthropic: breakpoints on the last tool, the system prompt and the newest message.
The hit rate is recorded per run from the usage fields, with an alert on regressions and a step-2 sanity check.

None of this is clever. It's the kind of discipline a database engineer applies to an index: work out what the cache keys on, then stop changing the key. Agents that follow it come out cheaper and faster, and they fail in fewer odd ways, because a context that only grows is also much easier to debug.