Step 1: name the job precisely
“Write marketing copy” is not a test. “Turn a 1,000-word customer interview into a 400-word case-study draft with no unsupported claims” is a test. Write the input format, output format, privacy constraints, budget, and what makes the result unusable.
Step 2: build a small ugly test set
Use five to ten real or safely anonymized examples: an ordinary case, a difficult case, a long case, a case with missing information, and a known failure case. A polished demo prompt tells you almost nothing about day-to-day performance.
Step 3: measure approved output
| Measure | What to record |
|---|---|
| Useful result | Did the output solve the job without changing the brief? |
| Correction time | Minutes before a human can approve it |
| Factual safety | Unsupported claims, wrong numbers, or invented sources |
| Total cost | Subscription, API usage, and human review |
| Speed | Time until a usable result, not first-token latency |
A simple scoring method
Score each result from 0 to 2 for accuracy, usefulness, instruction-following, and review effort. Keep the notes. The notes are more valuable than a single average because they tell you why a model failed and whether the failure is acceptable for the job.
Step 4: choose the workflow, not the leaderboard
Choose the model that produces the cheapest approved result for your actual process. If one model costs twice as much but cuts review from 20 minutes to 5, its invoice is not the relevant number.