The failure fine-tuning can address
Suppose a classifier understands the documents but keeps returning inconsistent labels, or a formatter misses the same output structure on repeated examples. Those are behavior problems. A well-curated set of examples may improve them. If the model simply lacks today’s internal facts, fine-tuning is the wrong fix; retrieval is usually the more direct path.
Before you train anything
- Write the task and success criteria.
- Build a representative evaluation set, including failures.
- Try a better prompt and explicit output schema.
- Check whether RAG or a tool call supplies the missing information.
- Measure the current human correction cost.
- Confirm that the examples are accurate, consistent, and legally usable.
Signs the investment may be justified
| Signal | Why it matters |
|---|---|
| The same behavior fails repeatedly | A stable pattern is more trainable than a moving target |
| You have many approved examples | Training quality depends on the examples |
| The task runs often enough | There must be a measurable payoff |
| You can rerun evals after changes | Fine-tuning creates a maintenance obligation |
| Prompting/RAG already reached a baseline | You can prove the incremental gain |
When not to fine-tune
- The source facts change weekly.
- You have fewer than a useful set of approved examples.
- No one can explain what a correct answer looks like.
- The task is rare or low-value.
- The team cannot maintain versioned evaluations and rollback plans.
The practical decision
Run a small baseline comparison first. If the tuned model does not reduce correction time or improve a measured success rate enough to pay for data preparation, training, and maintenance, keep the simpler system. The right answer is often better inputs and evaluation, not another model artifact.