Memory for Multimodal Agents

Text-only memory breaks the moment your agent sees an image. Multimodal agents need memory that spans modalities — and the design is different from text-only.

Most agent memory is text-only because most agents were text-only. The moment an agent handles images, audio, or video, the memory design has to span modalities — and a text-only store patched with captions loses everything the captions can't capture.

• **Cross-modal retrieval.** A text query retrieving relevant images, and vice versa.
• **Modality-specific storage.** Don't reduce an image to a caption and lose the visual detail.
• **Shared references.** A memory that refers to "the diagram" must work across text and image.
• **Unified retrieval.** One query that returns the right modality, not one per store.

Multimodal memory is also a story problem: the agent's memory has to support a coherent narrative across modalities — "I saw this diagram, you asked about it, here's what it showed." A memory that loses the cross-modal reference loses the story, and the agent sounds like it doesn't remember what it just saw.

• An image reduced to a caption, losing visual detail.
• A query that can't retrieve across modalities.
• References that break between text and image.
• Memory that "remembers" the caption but not the image.

• Store modality-specific representations (embeddings per modality) alongside shared references.
• Retrieve cross-modal via a shared embedding space or cross-encoder.
• Keep the raw modality available, not just a summarized version.
• Unify retrieval so one query returns the right modality.

Multimodal agents need memory that spans modalities — modality-specific storage, cross-modal retrieval, and shared references. Don't reduce images to captions; don't split memory into modality silos. The memory has to support the story, not just the text.