Knowledge / Field guide
Build a small RAG evaluation set before choosing a model
A good test collection tells you why an answer failed. Start with actual staff questions and a reviewer who knows where the answer lives.
2XD Studio · · 2 min read
Capture questions in the language people use
A buyer might ask whether a rush order needs a second signature. The policy might call this an expedited procurement exception. Include both phrasings, a misspelling, and a question with an irrelevant detail. Avoid writing every test by copying a sentence from the source document.
Create a compact answer key
For each question, record the acceptable answer, supporting passage, applicable role, and any exception. Include a few questions the library cannot answer. Set aside a group of cases that the team tuning the system does not use for day-to-day adjustments. Otherwise a promising improvement may only reflect memorized examples.
Score retrieval before prose
Check whether an acceptable passage appeared among the retrieved results. Then judge whether the final answer used it correctly. If the right passage never arrived, rewriting the answer prompt is unlikely to fix the miss. Compare results by question type so gains on easy lookups do not conceal worse performance on exceptions.
Read the error matrix
Suppose a new configuration improves exact-name searches but loses questions phrased in store-floor language. The average score may barely change. Look at the misses alongside the people affected: an improvement for managers may impose more manual searching on every shift. Add representative synonyms or better context, then retest both groups before declaring a winner.
Reader questions
How many questions do we need to start?
Enough to represent the decisions the pilot will support. A small reviewed set covering real failure modes is more useful than a large unverified list.
Who should grade answers?
Use a domain owner for correctness and an intended user for usability. Record disagreements rather than averaging them away.
Further reading Microsoft Learn: retrieval-augmented generation