2024 SelfDiscoverLargeLanguageModels
- (Zhou, Pujara et al., 2024) ⇒ Pei Zhou, Jay Pujara, Xiang Ren, Xinyun Chen, Heng-Tze Cheng, Quoc V. Le, Ed H. Chi, Denny Zhou, Swaroop Mishra, and Huaixiu Steven Zheng. (2024). “Self-Discover: Large Language Models Self-Compose Reasoning Structures.”
Subject Headings: SELF-DISCOVER.
Notes
- It introduces a self-discovery process where LLMs autonomously select and integrate various atomic reasoning modules (e.g., critical thinking, step-by-step analysis) into a coherent reasoning structure tailored to the specific task at hand.
- It can demonstrate performance improvements on challenging reasoning benchmarks (e.g., BigBench-Hard, MATH) with up to 32% enhancement over existing methods like Chain of Thought (CoT), and over 20% against CoT-Self-Consistency while requiring significantly less computational resources.
- It can suggest a universal applicability of the self-discovered reasoning structures across different LLM families (e.g., from PaLM 2-L to GPT-4), highlighting the adaptability of the framework.
- It can provide a more interpretable insight into task-solving, as the reasoning structures are explicit and reflect LLM's understanding of the task, compared to traditional optimized prompts.
- It can employ a two-stage process for reasoning structure discovery and application, where the first stage identifies a task-specific structure, and the second stage applies this structure to solve individual task instances.
- It can showcase the efficiency of SELF-DISCOVER, which outperforms other inference-heavy methods by requiring significantly fewer computational resources for similar or better performance.
- It can align LLM reasoning patterns with human reasoning, as shown by the commonalities between self-discovered reasoning structures and human problem-solving approaches, suggesting the potential for enhancing human-AI collaboration.
Cited By
Quotes
Abstract
We introduce SELF-DISCOVER, a general framework for LLMs to self-discover the task-intrinsic reasoning structures to tackle complex reasoning problems that are challenging for typical prompting methods. Core to the framework is a self-discovery process where LLMs select multiple atomic reasoning modules such as critical thinking and step-by-step thinking, and compose them into an explicit reasoning structure for LLMs to follow during decoding. SELF-DISCOVER substantially improves GPT-4 and PaLM 2's performance on challenging reasoning benchmarks such as BigBench-Hard, grounded agent reasoning, and MATH, by as much as 32% compared to Chain of Thought (CoT). Furthermore, SELF-DISCOVER outperforms inference-intensive methods such as CoT-Self-Consistency by more than 20%, while requiring 10-40x fewer inference compute. Finally, we show that the self-discovered reasoning structures are universally applicable across model families: from PaLM 2-L to GPT-4, and from GPT-4 to Llama2, and share commonalities with human reasoning patterns.
References
;
| Author | volume | Date Value | title | type | journal | titleUrl | doi | note | year | |
|---|---|---|---|---|---|---|---|---|---|---|
| 2024 SelfDiscoverLargeLanguageModels | Jay Pujara Quoc V. Le Xiang Ren Heng-Tze Cheng Denny Zhou Ed H. Chi Swaroop Mishra Xinyun Chen Pei Zhou Huaixiu Steven Zheng | Self-Discover: Large Language Models Self-Compose Reasoning Structures | 2024 |