Busted: Uncovering Reranker Blind Spots in RAG Pipelines

You add a cross-encoder reranker to your RAG pipeline, measure answer quality on a test set, see a marginal improvement on 3 of 8 questions, and ship it. Six weeks later your p99 retrieval latency has climbed 200ms per query and you're paying Cohere API costs on every call. Nobody has revisited the