Agent Memory: Patterns That Actually Work in Production

Every agent framework promises memory.

"Persistent memory across sessions." "Long-term learning." "Agents that remember."

Then you try to use it in production.

Your agent forgets critical context mid-conversation. Memory bloats until responses slow to a crawl. The agent "remembers" things that never happened. Retrieval pulls irrelevant memories while missing obvious ones.

I've built memory systems for dispute resolution agents handling 15,000+ cases, marketing AI that maintains brand voice across thousands of interactions, and financial advisors tracking complex client portfolios.

Here's what I've learned: agent memory is a systems problem, not a feature toggle. The frameworks give you primitives. Making them work requires architecture.

First, let's understand why this is harder than it looks.

After building several production systems, I've converged on a layered architecture:

Each layer serves a different purpose. Each has different storage, retrieval, and lifecycle characteristics.

Working memory is what's in the prompt right now. It's the agent's immediate awareness.

The Pattern: Sliding Window + Summarization

class WorkingMemory:
    def __init__(self, max_turns: int = 10, max_tokens: int = 4000):
        self.max_turns = max_turns
        self.max_tokens = max_tokens
        self.turns = []
        self.summary = ""

    def add_turn(self, role: str, content: str):
        self.turns.append({"role": role, "content": content})

        # If exceeding limits, summarize older turns
        if len(self.turns) > self.max_turns:
            self._compress()

    def _compress(self):
        # Summarize oldest turns
        old_turns = self.turns[:5]
        new_summary = llm.summarize(old_turns)

        # Update summary and remove old turns
        self.summary = f"{self.summary}\n{new_summary}"
        self.turns = self.turns[5:]

Key insight: Don't just truncate old messages—summarize them. You lose detail but preserve the important bits.

Short-term memory bridges sessions. It's what the agent remembers from earlier today, yesterday, this week.

The Pattern: Time-Decayed Cache

class ShortTermMemory:
    def __init__(self, redis_client, decay_hours: int = 72):
        self.redis = redis_client
        self.decay_hours = decay_hours

    def store(self, user_id: str, key: str, value: dict):
        memory_key = f"stm:{user_id}:{key}"
        value["timestamp"] = time.time()
        value["access_count"] = 0
        self.redis.setex(
            memory_key,
            timedelta(hours=self.decay_hours),
            json.dumps(value)
        )

    def retrieve(self, user_id: str, key: str) -> dict | None:
        data = self.redis.get(f"stm:{user_id}:{key}")

        if data:
            value = json.loads(data)
            # Boost TTL on access (frequently accessed = more important)
            value["access_count"] += 1
            self.redis.setex(...)  # Extend TTL
            return value
        return None

Key insight: Use TTLs and access patterns to naturally decay irrelevant memories. Memories that aren't accessed fade away.

Long-term memory is persistent knowledge. User profiles, learned preferences, entity relationships.

The Pattern: Structured + Unstructured Hybrid

Long-term memory needs both structured storage (for reliable retrieval) and unstructured storage (for semantic search).

Key insight: Use the graph for explicit relationships and structured queries. Use vectors for semantic similarity. You need both.

Episodic memory stores specific past interactions—memorable conversations, key decisions, important events.

The Pattern: Selective Storage with Importance Scoring

You can't store every interaction. You need to decide what's worth remembering.

Key insight: Not all interactions deserve to be memories. Score importance and be selective.

Memory adds latency. Here's how to manage it: