Evaluating RAG retrieval with adversarial questions

Retrieval-augmented generation is only as good as its retrieval.

We built an adversarial evaluation set where the correct chunk is always present but surrounded by near-duplicates and plausible distractors. The results show how quickly answer quality degrades when ranking is even slightly off.

This experiment is a cautionary tale for anyone shipping RAG without a rigorous retrieval eval.