OSWorld AI Agent Framework

From GM-RKB
Jump to navigation Jump to search

A OSWorld AI Agent Framework is a AI agent software platform that facilitates the development, testing, and benchmarking of AI agents within a simulated real computer environment.

References

2024

  • https://os-world.github.io/
    • NOTES
      1. **Supports a Wide Range of Operating Systems**: OSWorld operates across various platforms like Ubuntu, Windows, and macOS, providing a comprehensive environment for testing AI agents in real-world settings similar to those used by professionals and consumers .
      2. **Incorporates Extensive Benchmark Suite**: The platform includes a benchmark suite of 369 real-world tasks, ranging from web and desktop applications to OS file I/O operations, allowing for detailed and rigorous testing of AI agent capabilities .
      3. **Facilitates Reproducible and Reliable Evaluation**: Each task within OSWorld is accompanied by a detailed initial state setup and a custom execution-based evaluation script, ensuring that tests are both reproducible and reliable, essential for the iterative improvement of AI agents .
      4. **Enables Real-Time Interactive Learning**: The platform supports interactive learning, which allows AI agents to learn from actions and improve over time, a critical feature for agents expected to operate dynamically in varied legal environments .
      5. **Open for Collaboration and Contribution**: OSWorld encourages open-source collaboration, inviting developers to contribute to its suite of tools and tasks, which helps in broadening the scope and applicability of the platform to new domains and challenges .

2024

  • (Xie et al., 2024) ⇒ Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua et al. (2024). “OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments.” arXiv preprint arXiv:2404.07972
    • ABSTRACT: Autonomous agents that accomplish complex computer tasks with minimal human interventions have the potential to transform human-computer interaction, significantly enhancing accessibility and productivity. However, existing benchmarks either lack an interactive environment or are limited to environments specific to certain applications or domains, failing to reflect the diverse and complex nature of real-world computer use, thereby limiting the scope of tasks and agent scalability. To address this issue, we introduce OSWorld, the first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across various operating systems such as Ubuntu, Windows, and macOS. OSWorld can serve as a unified, integrated computer environment for assessing open-ended computer tasks that involve arbitrary applications. Building upon OSWorld, we create a benchmark of 369 computer tasks involving real web and desktop apps in open domains, OS file I/O, and workflows spanning multiple applications. Each task example is derived from real-world computer use cases and includes a detailed initial state setup configuration and a custom execution-based evaluation script for reliable, reproducible evaluation. Extensive evaluation of state-of-the-art LLM/VLM-based agents on OSWorld reveals significant deficiencies in their ability to serve as computer assistants. While humans can accomplish over 72.36% of the tasks, the best model achieves only 12.24% success, primarily struggling with GUI grounding and operational knowledge. Comprehensive analysis using OSWorld provides valuable insights for developing multimodal generalist agents that were not possible with previous benchmarks. Our code, environment, baseline models, and data are publicly available at this https URL.
Retrieved from "http://www.gabormelli.com/RKB/index.php?title=OSWorld_AI_Agent_Framework&oldid=897368"