What is RLHF (reinforcement learning from human feedback)?

RLHF, or reinforcement learning from human feedback, is a training technique that teaches an LLM to give answers people actually find helpful, honest, and safe — not just statistically likely ones. It's a big part of why modern chatbots feel cooperative and on-your-side rather than like raw text predictors that finish your sentence.

How does RLHF work?

The idea is to let human preference, rather than raw text, do the teaching. It usually runs in a few steps:

1. Start with a pretrained model that produces fluent text but has no sense of what makes an answer good.
2. Have people compare pairs of its answers and pick which one is better, building up a dataset of preferences.
3. Train a second model, called a reward model, on those preferences so it can score any answer the way a human would.
4. Fine-tune the LLM to produce answers that earn high scores — essentially practicing until its responses match human taste.

Why is RLHF needed?

This closes a gap that pretraining leaves open. A pretrained model knows a lot, but it has no sense of what makes a response good: it might be rude, evasive, or confidently wrong. RLHF aligns its behavior with human expectations for tone, helpfulness, and refusal of harmful requests — turning raw knowledge into something usable.

What are RLHF's limitations?

It isn't perfect. It can make models overly agreeable — telling you what you want to hear instead of what's true — and it reflects the judgments of whoever gave the feedback, so their blind spots can carry through. Some newer approaches swap human raters for AI-generated feedback to cut the cost and slowness of collecting human labels, but the core loop — score answers, then train the model toward the ones that score well — stays the same.

Even so, RLHF was a pivotal step in turning powerful-but-unruly language models into assistants people can comfortably use, and variants of it are now standard practice.