Agent evals that run manually catch regressions after they ship. Eval-driven development needs the eval in CI — so every change is gated, not just the ones you remember to eval.
An eval you run by hand is an eval that doesn't run on every change. The discipline that makes evals actually gate regressions is putting them in CI, so the change that breaks the agent doesn't ship — the same way a broken test doesn't ship.
First Things: The Eval in CI
Begin with the end in mind: every change to the agent runs the eval suite, and a regression fails the build. The eval in CI is the mechanism that makes eval-driven development real — without it, the eval is a ritual; with it, the eval is a gate.
- The eval suite runs on every change.
- A regression fails the build.
- The change doesn't ship until the eval passes.
Process Measurement: What CI Catches
The eval in CI catches what manual evals miss:
- A prompt change that broke the long tail.
- A model upgrade that improved average and broke edges.
- A retrieval change that regressed precision.
- A dependency change that changed agent behavior.
Each is a regression that would have shipped without the CI gate.
What to Build
- An eval suite that runs headless, fast, and reproducibly.
- A CI job that runs it on every change.
- A failure that blocks the build.
- A retention of results so regressions are visible over time.
What to Watch
- Eval latency — a slow eval suite becomes a CI bottleneck; keep it fast.
- Flakiness — a flaky eval is worse than none; fix flakes before trusting the gate.
- The eval set drifting from real usage — refresh it as the product changes.
What to Refuse
- An eval suite that's too slow to run on every change.
- A flaky eval that fails randomly and trains people to ignore failures.
- An eval set that doesn't cover the failure modes that matter.
- A CI gate that people bypass because it's noisy.
Conclusion
Agent evals in CI are what make eval-driven development real. Run the suite on every change, fail the build on regression, and keep the eval fast, stable, and current. The gate that catches regressions before users do is the one that ships — and the one that doesn't is the one that ships regressions.
About FACTA
FACTA helps startups and growth-stage teams turn AI into production systems that keep running — not demos that impress once.
We design the architecture around the parts that actually break under real usage: tooling you own, credentials you control, failover, cost controls, observability. The boring infrastructure that keeps a system alive after launch.
Led by Matías Baglieri and Carolina Fogliato, we focus on one thing:
AI leadership that builds. Not just advises.
Tell us whether your agent evals run in CI today.
We'll tell you what it would take to gate every change. See eval-driven RAG for the eval-side foundation.
Explore AI Automation