Caching for Agents Stop Recomputing the Same Thing

Agents recompute the same deterministic steps on every request — retrieval, tool calls, sub-agent outputs — and pay for them every time. Caching is the cheap win most pipelines skip.

A surprising share of agent latency and cost is recomputation: the same retrieval, the same tool call, the same sub-agent output, recomputed per request. Caching turns those into a one-time cost, and the saving repeats every request.

Caching is a cashflow decision: a deterministic step computed once and reused is a recurring saving. Across a workflow that runs many times, the saving compounds — and the cost of the cache is one-time. The pipeline that caches pays less every request and ships faster every request.

• Compute once, reuse many times.
• Latency saving per request.
• Cost saving per request.

Cache the deterministic — the things that don't change between requests:

• Retrieval results for the same query (semantic cache).
• Tool outputs for the same inputs.
• Sub-agent outputs for the same sub-task.
• Anything computed deterministically.

Don't cache the nondeterministic — model generation that should vary per request, anything that depends on changing state.

• Cache invalidation — when the underlying data changes, the cache must too.
• Cache keys — semantic caching needs a key that captures the intent, not the exact text.
• Staleness — a stale cache is worse than no cache if it serves wrong answers.

• Caching nondeterministic outputs.
• Caching without an invalidation strategy.
• Caching everything (cache the hot path, not the cold).

Caching is the cheap win most agent pipelines skip. Cache the deterministic — retrieval, tool calls, sub-agent outputs — and the saving repeats every request. The pipeline that caches pays less, ships faster, and compounds the saving.