World of Bits: An Open-Domain Platform for Web-Based Agents
(Aug. 2017, ICML 2017)
(Aug. 2017, ICML 2017)
Awesome GUI Agent
💻 A curated list of papers and resources for multi-modal Graphical User Interface (GUI) agents.
This page lists names, links and short descriptions. The original list on GitHub is the source and belongs to its authors.
Datasets / Benchmarks
A Unified Solution for Structured Web Data Extraction
(Jul. 2011, SIGIR 2011)
Reinforcement Learning on Web Interfaces using Workflow-Guided Exploration
(Feb. 2018, ICLR 2018)
Mapping Natural Language Instructions to Mobile UI Action Sequences
(May. 2020, ACL 2020)
WebSRC: A Dataset for Web-Based Structural Reading Comprehension
(Jan. 2021, EMNLP 2021)
Language Models can Solve Computer Tasks
(Mar. 2023)
AssistGUI: Task-Oriented Desktop Graphical User Interface Automation
(Dec. 2023, CVPR 2024)
In 2 lists
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
(Jan. 2024, ACL 2024)
Benchmarking Mobile Device Control Agents across Diverse Configurations
(Apr. 2024, ICLR 2024)
MONDAY: Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents
(May., 2025, CVPR 2025)
Models / Agents
Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control
(Jun. 2023, ICLR 2024)
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis
(Jul. 2023, ICLR 2024)
CogAgent: A Visual Language Model for GUI Agents
(Dec. 2023, CVPR 2024)
WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models
Vision-enabled benchmark for real-world website interaction with large multimodal models.
In 2 lists
In 2 lists
SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models
(May. 2023, NeurIPS 2023)
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
(Jan. 2024, ACL 2024)
GPT-4V(ision) is a Generalist Web Agent, if Grounded
(Jan. 2024, ICML 2024)
Dual-View Visual Contextualization for Web Navigation
(Feb. 2024, CVPR 2024)
Visual Grounding for User Interfaces
(NAACL 2024)
Octopus v4: Graph of language models
(Apr., 2024)
Enhancing Mobile "How-to" Queries with Automated Search Results Verification and Reranking
(Apr. 2024, SIGIR 2024)
Identifying User Goals from UI Trajectories
(Jun. 2024)
OmniParser for Pure Vision Based GUI Agent
(Aug. 2024)
Agent Workflow Memory
(Sep. 2024)
CogAgent v2
(Dec. 2024)
Lightweight Neural App Control
(ICLR 2025)
Surveys
GUI Agents: A Survey
(Dec. 2024)
Projects
GPT-4V-Act: AI agent using GPT-4V(ision) for web UI interaction
AI agent using GPT-4V(ision) capable of using a mouse/keyboard to interact with web UI
In 2 lists
gpt-computer-assistant
GPT Computer Assistant is an unofficial app that brings ChatGPT functionality to Windows and Linux, allowing for screen reading, microphone use, system audio interaction, clipboard management, script execution, and more github
In 3 lists
Mobile-Agent: The Powerful Mobile Device Operation Assistant Family
Powerful GUI agent family for autonomous mobile device control. Multimodal agent framework designed to operate smartphone apps through visual UI perception and reasoning. MIT licensed.
In 4 listsDetails
OpenUI
与 v0.dev 相似，但是是基于 Prompt 构建 UI 组件
In 2 lists
NatBot
Drive a browser with GPT-3
In 2 lists
Multion
MultiOn utilizes AI to automate actions within web browsers, such as form filling, data retrieval, and executing web searches, mimicking human interaction but without manual input, facilitated through a Chrome extension and API for developers website
In 3 lists
LaVague: Large Action Model Framework to Develop AI Web Agents
Automate automation with Large Action Model framework.
In 2 lists
OpenAdapt: AI-First Process Automation with Large Multimodal Models
Visual Language (VLMs)) Models
In 2 lists
Open Interpreter
Open Interpreter is a coding agent enabling language models to execute code locally, facilitating natural-language interaction with your computer's capabilities, overcoming limitations of hosted solutions like internet access and package restrictions. It features interactive and programmatic…
In 9 listsDetails
Safety
Related Repositories
Related lists in Computer Science
See category94
Table of Contents
hesreallyhim/awesome-claude-code
A hand-picked collection of the finest of resources for the most awesome of agents, Claude Code, the undisputed champion of coding companions, from the unstoppable team…
94
Awesome Agent Skills
VoltAgent/awesome-agent-skills
A curated collection of 1000+ agent skills from official dev teams and the community, compatible with Claude Code, Codex, Gemini CLI, Cursor, and more.
93
Awesome Machine Learning
josephmisiti/awesome-machine-learning
A curated list of awesome Machine Learning frameworks, libraries and software.
92
Awesome Production Machine Learning
EthicalML/awesome-production-machine-learning
A curated list of awesome open source libraries to deploy, monitor, version and scale your machine learning
92
AWESOME DATA SCIENCE
academic/awesome-datascience
:memo: An awesome Data Science repository to learn and apply for real world problems.
91
Static Analysis
analysis-tools-dev/static-analysis
⚙️ A curated list of static analysis (SAST) tools and linters for all programming languages, config files, build tools, and more. The focus is on tools which improve…