Direct Preference Optimization (DPO)

A Direct Preference Optimization (DPO) is an LLM instruction fine-tuning algorithm that aligns language models with human preferences by formulating and maximizing a reward function derived directly from these preferences.

• Context:
  • It can (typically) involve constructing a reward function directly based on human preference data.
  • ...
• Example(s):
  • as proposed in (Rafailov et al., 2023).
  • ...
• Counter-Example(s):
  • Reinforcement Learning from Human Feedback.
• See: Model Optimization, Language Model Alignment, Human-Centered AI.

---

---

References

2024

• (Wikipedia, 2024) ⇒ https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback#Alternatives Retrieved:2024-1-11.
  • An alternative to RLHF called Direct Preference Optimization (DPO) was described in 2023. Like RLHF, it is used to improve pre-trained large language models using human-generated preference data. Unlike RLHF, it does not train an intermediate reward model and does not use reinforcement learning; instead, it formulates a reward function based on the human preferences and directly trains the large language model to maximize this reward.

2023

• (Rafailov et al., 2023) ⇒ Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. (2023). “Direct Preference Optimization: Your Language Model is Secretly a Reward Model.” doi:10.48550/arXiv.2305.18290
  • NOTE:
    • It introduces Direct Preference Optimization (DPO), a novel approach for aligning language models with human preferences, as a simpler alternative to the complex and often unstable reinforcement learning from human feedback (RLHF).
    • It demonstrates a unique mathematical insight, showing that for any given language model, there exists a specific reward function for which the model is optimal, eliminating the need for a separately represented reward function.
    • It simplifies aligning instruction-tuned LLM models to human preferences (by requiring only the language model transformer for training, since the reward function is implicitly defined).
    • It can be computationally lighter and easier to implement than RLHF (which involves training two transformer networks and is sensitive to hyperparameter choices).

---