2024 ManagingExtremeAIRisksAmidRapid
- (Bengio, Hinton et al., 2024) ⇒ Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Trevor Darrell, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, Gillian Hadfield, Jeff Clune, Tegan Maharaj, Frank Hutter, Atılım Güneş Baydin, Sheila McIlraith, Qiqi Gao, Ashwin Acharya, David Krueger, Anca Dragan, Philip Torr, Stuart Russell, Daniel Kahneman, Jan Brauner, and Sören Mindermann. (2024). “Managing Extreme AI Risks Amid Rapid Progress.” In: Science. 10.1126/science.adn0117 doi: 10.1126/science.adn0117
Subject Headings: AI Risk, AI Safety, Frontier AI System.
Notes
- The paper highlights the rapid progress in AI development and the increasing focus on creating generalist AI systems capable of autonomously acting and pursuing goals.
- The paper emphasizes the potential risks associated with advanced AI, including large-scale social harms, malicious uses, and the irreversible loss of human control over autonomous AI systems.
- The paper identifies a significant lag in AI safety research compared to the rapid advancements in AI capabilities and stresses the need for a more balanced approach.
- The paper calls for a reorientation of technical research and development (R&D) to prioritize AI safety, highlighting the necessity for dedicated efforts to ensure the ethical use of AI.
- The paper advocates for proactive and adaptive governance mechanisms to address the risks of AI, suggesting that current initiatives are insufficient to manage the potential dangers.
- The paper outlines specific R&D challenges that need breakthroughs to ensure AI safety, including oversight, robustness, interpretability, and inclusive AI development.
- The paper recommends the allocation of at least one-third of AI R&D budgets towards addressing AI safety and ethical use, proposing various incentives and support mechanisms.
- The paper stresses the importance of establishing national and international governance frameworks that can keep pace with AI advancements, drawing lessons from other safety-critical technologies.
- The paper highlights the necessity of technically skilled institutions for AI oversight and suggests the development of safety cases and comprehensive risk assessments for frontier AI systems.
Cited By
Quotes
Abstract
Artificial intelligence (AI) is progressing rapidly, and companies are shifting their focus to developing generalist AI systems that can autonomously act and pursue goals. Increases in capabilities and autonomy may soon massively amplify AIâs impact, with risks that include large-scale social harms, malicious uses, and an irreversible loss of human control over autonomous AI systems. Although researchers have warned of extreme risks from AI (1), there is a lack of consensus about how to manage them. Societyâs response, despite promising first steps, is incommensurate with the possibility of rapid, transformative progress that is expected by many experts. AI safety research is lagging. Present governance initiatives lack the mechanisms and institutions to prevent misuse and recklessness and barely address autonomous systems. Drawing on lessons learned from other safety-critical technologies, we outline a comprehensive plan that combines technical research and development (R&D) with proactive, adaptive governance mechanisms for a more commensurate preparation.
References
;