Create a test set with consequences
Include a normal request, an ambiguous request, a request with missing data, a permission failure, a tool timeout, and a request that should be refused. Testing only the happy path tells you whether the demo works, not whether the system is safe.
Record the complete trace
- User request and permissions.
- Plan or intermediate reasoning exposed by the product.
- Every tool call and its arguments.
- Returned tool data and its age.
- Final action or answer.
- Human approvals, retries, and total cost.
Pass criteria
| Area | A passing test should show |
|---|---|
| Task | The agent reaches the intended outcome |
| Tools | Correct tool, valid arguments, no unnecessary calls |
| Safety | High-risk actions stop for approval |
| Uncertainty | Missing evidence produces a clear fallback |
| Operations | Bounded cost, time, retries, and logs |
Start in shadow mode
Let the agent propose actions while a person performs or approves them. Compare the proposal with the human result over repeated cases. Remove approval gates only after the failure patterns are understood and the business can tolerate the remaining risk.