DPO vs PPO: Direct Preference Optimization Wins Post-Training
Direct Preference Optimization (DPO) collapses the three-network, reinforcement-learning pipeline behind classic RLHF, a reward model plus PPO, into a single supervised-style loss trained directly on preference pairs. It has become the default preference-alignment method across most open-weight model releases, while frontier labs increasingly pair it with on-policy RL only for reasoning tasks that carry verifiable rewards. The bigger signal is that preference optimization is consolidating into a modular pipeline rather than any one algorithm winning outright. For PMs, which alignment method a vendor or in-house team actually runs says a lot about whether they're optimizing for stability and speed, or holding onto flexibility for the next hard problem.