Edinburgh-trained IR researcher Sarah Chen highlights how a new metric called "Bits over Random" (BoR) exposes a critical flaw in how we measure RAG systems. While traditional metrics like Success@K and recall focus on finding relevant information, BoR measures whether retrieval is actually selective or just stuffing context windows with more material. The research shows that systems can achieve 99% success rates while performing barely better than random selection—a phenomenon that explains why many production RAG systems look good on dashboards but produce diffuse, unreliable agent behavior.
This matters because most RAG teams are optimizing for the wrong thing. Classic IR thinking—did we find relevant chunks, did recall improve—works fine for search engines where humans filter results. But LLM agents must process everything you give them, making context pollution a real performance killer. When you crank up recall by retrieving more chunks, you often drag along weakly relevant material that dilutes the model's attention and degrades reasoning quality.
What's particularly striking is how this research validates what many practitioners have felt but couldn't articulate: retrieval that looks excellent on paper can behave like noise in production. The BoR metric provides a mathematical framework for something builders have been debugging through intuition—that more context isn't always better context.
For developers, this research suggests rethinking your evaluation stack. Instead of just measuring whether you found relevant information, start measuring how much irrelevant material comes with it. Consider retrieval selectivity as a first-class metric alongside traditional recall measures. Your agents will thank you with more focused, reliable behavior—even if your dashboard numbers look less impressive.