2024 OmniParserforPureVisionBasedGUI

From GM-RKB
(Redirected from Lu et al., 2024)
Jump to navigation Jump to search

Subject Headings: OmniParser, Vision-based AI Model, Screen Parsing, Multimodal AI, UI Element Semantics, Interactivity Detection, Functional Semantics Extraction, Cross-Platform AI Agents, Benchmarks for Vision Models.

Notes

Cited By

Quotes

Abstract

The recent success of large vision language models shows great potential in driving the agent system operating on user interfaces. However, we argue that the power of multimodal models like GPT-4V as a general agent on multiple operating systems across different applications is largely underestimated due to the lack of a robust screen parsing technique capable of: 1) reliably identifying interactable icons within the user interface, and 2) understanding the semantics of various elements in a screenshot and accurately associating the intended action with the corresponding region on the screen. To fill these gaps, we introduce OmniParser, a comprehensive method for parsing user interface screenshots into structured elements, which significantly enhances the ability of GPT-4V to generate actions that can be accurately grounded in the corresponding regions of the interface. We first curated an interactable icon detection dataset using popular webpages and an icon description dataset. These datasets were utilized to fine-tune specialized models: a detection model to parse interactable regions on the screen and a caption model to extract the functional semantics of the detected elements. OmniParser significantly improves GPT-4V's performance on the ScreenSpot benchmark. And on Mind2Web and AITW, OmniParser with screenshot-only input outperforms the GPT-4V baselines requiring additional information outside of the screenshot.

References

;

 AuthorvolumeDate ValuetitletypejournaltitleUrldoinoteyear
2024 OmniParserforPureVisionBasedGUIYelong Shen
Ahmed Awadallah
Jianwei Yang
Yadong Lu
OmniParser for Pure Vision Based GUI Agent10.48550/arXiv.2408.002032024