Reward-model RLHF versus direct preference optimization

Walk through a classic RLHF pipeline: supervised fine-tuning, a reward model trained on preference pairs, then policy optimization (PPO) with a KL penalty to the reference model. Then explain how DPO reaches a similar goal with no reward model and no sampling loop, what its loss compares, and what role the reference model still plays. Compare cost, stability and failure modes, and say when you would pick each.