Agno Performance Optimization: From 2 Seconds to 200ms

First, let's talk about why we cared about 200ms specifically.

For our dispute resolution system, users are in emotional situations. They're frustrated. They want answers. Every second of latency compounds that frustration.

But it's not just UX. At scale, latency compounds:

That's 71 hours of user time saved. Per year. For one system.

Let's start with what we were working with. Here's where the time went:

GPT-4 was the biggest chunk, but we couldn't just "make GPT-4 faster." We needed to optimize everything we controlled.

The first win was obvious once we looked at it. Our agent called four tools sequentially:

But these tools don't depend on each other. They can run in parallel.

agent = DisputeAnalysisAgent(
    tool_execution_mode="parallel"  # Enable parallel tools
)
# Total: 180ms (longest tool)

Here's the counterintuitive one: not every task needs GPT-4.

Our dispute analysis had multiple sub-tasks:

• • Case classification (simple)
• • Entity extraction (medium)
• • Legal reasoning (complex)
• • Confidence scoring (simple)

GPT-4 is overkill for classification. A fine-tuned small model is faster AND cheaper.

model = ModelRouter({
    "classification": "gpt-3.5-turbo",     # Fast, cheap
    "extraction": "claude-3-haiku",         # Good at structure
    "reasoning": "gpt-4",                   # Complex tasks only
    "scoring": "gpt-3.5-turbo",            # Fast, cheap
})

We were passing way too much context. For a complex case, that could be 15,000+ tokens of context.

Problem: More tokens = more latency. GPT-4's time-to-first-token scales with input size.

def build_optimized_context(case: Case, task: str) -> str:
    # Always include core identifiers
    base = f"Case: {case.id} | Type: {case.dispute_type}"

    if task == "classification":
        return base + f"\nDescription: {case.description[:500]}"

    elif task == "reasoning":
        # Only for complex reasoning do we include more
        relevant_docs = retrieve_relevant_chunks(case, top_k=3)
        return base + f"\nRelevant Sections: {relevant_docs}"

    return base

# Before: 10,000+ tokens
# After: 200-800 tokens depending on task

For many tasks, we don't need the full response. We can start processing as soon as we have enough.

async def get_classification(case: Case) -> str:
    async for chunk in agent.stream(prompt):
        # Check if we have enough to classify
        if classification := try_parse_classification(chunk):
            return classification  # Exit early

    # Fallback: parse complete response
    return parse_classification(chunk)

For classification tasks, we typically exit after 30-40% of the response is generated.

Many inputs are repeated or similar. Caching prevents redundant computation.

This one's boring but impactful. New connection = 100-200ms overhead.

http_client = httpx.Client(
    http2=True,
    limits=httpx.Limits(
        max_connections=100,
        max_keepalive_connections=20,
        keepalive_expiry=30
    )
)

agno.configure(
    http_client=http_client,
    connection_pool_size=100
)

If you're doing anything synchronous in your agent pipeline, you're leaving performance on the table.

async def process_case(case: Case) -> Result:
    # Independent steps run in parallel
    classification, entities = await asyncio.gather(
        classify(case),
        extract_entities(case)
    )

    # Dependent step runs after
    analysis = await analyze(case, classification, entities)

    return Result(classification, entities, analysis)

Wait—that's 455ms, not 200ms. Right. For the average case.

If you're optimizing Agno performance, here's the priority order:

You don't need all of these. Implement 1-3 and you'll see dramatic improvement.

A word of caution: premature optimization is still a thing.

Don't optimize if:

• • Your latency is already acceptable for your use case
• • You're still iterating on agent behavior
• • You don't have production traffic yet
• • The complexity cost outweighs the latency benefit

Get it working first. Get it correct. Then make it fast.

Agno is fast. But "fast" is a starting point, not a destination.

The patterns here—model routing, parallel execution, caching, context optimization—aren't Agno-specific. They're general principles for building performant AI systems.

2 seconds to 200ms isn't magic. It's methodology.