Evaluating the Answers of a RAG System

You changed the chunking, switched embedding models, and added a reranker. The system seems better. Then a user reports a regression on a question that worked last week. You can neither confirm nor rule it out, because your only validation method is typing four questions you know by heart and finding the answers satisfactory. You don't know which of your three changes helped, which one hurt, or whether the average improved.

This is the default situation for almost every RAG project, and it's what makes them stall. A RAG system has two stages that fail for different reasons and are fixed by different means: retrieval and generation. A single global metric blends them, so it never tells you what to fix. The minimum discipline is to measure both separately, on a frozen question set, before every change.

Building the Reference Set

Aim for 50 to 150 questions. Below 40, statistical noise exceeds the differences you're trying to measure. Beyond 200, the annotation cost becomes prohibitive without a proportional gain.

Each entry holds the question, the identifier of the document that contains the answer, and a short reference answer. The source field is the most important one: it's what makes recall computable automatically.