Embedding Model Selection Benchmarks Lie

The embedding model that tops the leaderboard isn't the one that wins on your data. Leaderboards measure average quality on public datasets; you need the delta on your corpus.

Embedding model selection is where benchmark-driven decisions mislead. A model that's best on average can be worse than a cheaper model on your specific domain — and the only way to know is to eval on your data.

• **Your data.** Eval on your corpus, not the leaderboard's.
• **Your queries.** Eval on the queries your users actually run.
• **Your cost and latency.** A bigger model that wins by a point may cost 5x.
• **Your language and domain.** A general model may underperform a domain one.

The leaderboard gives you average quality. The decision needs the delta: how much better is model A than model B on your eval, at what cost and latency. A 1-point leaderboard win at 5x the cost is usually the wrong choice; a 5-point win at the same cost is usually the right one.

• Pick 3-5 candidates, including a cheap baseline.
• Eval each on your eval set (retrieval recall and precision).
• Compare on quality, cost, and latency — not quality alone.
• Choose the cheapest model that meets your quality bar.

• Choosing the leaderboard leader without evaluating on your data.
• Choosing a bigger model for a marginal quality win.
• Ignoring cost and latency in the decision.
• Skipping the cheap baseline that might be enough.

Embedding model selection is a decision about the delta on your data, not the average on a leaderboard. Eval candidates on your corpus, compare on quality-cost-latency, and choose the cheapest model that meets the bar.