Build a golden set
Start with examples that represent the work: normal cases, edge cases, ambiguous inputs, and examples that previously failed. Keep the expected outcome and why it is acceptable. A test set that contains only easy prompts will reward the wrong system.
Score more than the text
- Task success: did it solve the user’s actual job?
- Factuality: did it invent claims, sources, or numbers?
- Instruction following: did it respect the format and constraints?
- Human correction: how many minutes before approval?
- Operational fit: cost, latency, rate limits, and privacy?
Separate the failures
| Failure | Likely place to investigate |
|---|---|
| The right document was never found | Ingestion, chunking, embeddings, or retrieval |
| The source was found but ignored | Prompt, context ordering, or model behavior |
| The answer followed the source but missed the user’s need | Task definition or output contract |
| The output is useful but too expensive | Model choice, routing, caching, or workflow design |
Run the comparison fairly
Use the same inputs, tool permissions, prompt intent, and success criteria. Record model version, date, settings, and cost. If the product changes one variable between runs, mark the comparison as exploratory rather than conclusive.