version: "1.0.0"

CogAgent: A Visual Language Model for GUI Agents

One-line decision

Use this skill when you want to create training data for a VLM that can understand and interact with graphical user interfaces (GUIs). Avoid it when you do not need GUI understanding or agent capability.

Skill metadata

• Skill type: gui-interaction-data
• Paper kind: operational-method
• Actionability: high
• Evidence quality: full_paper

Goal

Create training data for a VLM agent that understands graphical user interfaces (GUIs) and can navigate them through visual understanding, enabling GUI automation.

Problem signature

• Modality: GUI screenshots with interaction instructions and grounding.
• Data state: GUI screenshots with annotated interaction data.
• Scale regime: curated GUI interaction instruction data.
• Model requirement: CogVLM with high-resolution GUI processing.

Use when

• You want a VLM that understands GUIs.
• You need GUI navigation and interaction capability.
• You can collect GUI interaction data.

Do not use when

• GUI understanding is not needed.
• You only need natural image understanding.
• API-based automation is preferred over visual.

Required inputs

• gui_screenshots: Screenshots of various GUIs (web, mobile, desktop).
• interaction_annotations: Annotated interaction sequences on GUIs.
• grounding_data: Element locations in GUI screenshots.

Optional inputs

• accessibility_tree: DOM or accessibility tree for verification.

Outputs

• gui_instruction_data: GUI interaction instruction data.
• cogagent_model: VLM agent for GUI understanding.

Assumptions and prerequisites

• VLMs can learn to understand GUI layouts.
• High resolution is critical for GUI text and element recognition.
• Interaction data teaches GUI navigation.

Procedure

1. Collect GUI screenshots

Action: Capture diverse GUI screenshots (web, mobile, desktop). Why: Diverse GUIs ensure broad understanding. Note: See paper for details.

1. Annotate interactions

Action: Create instruction-action pairs for GUI tasks. Why: Interaction data teaches navigation. Note: See paper for details.

1. Add grounding annotations

Action: Annotate element locations for grounding. Why: Grounding enables precise interaction. Note: See paper for details.

1. Train CogAgent

Action: Train CogVLM with GUI-specific data. Why: Specialized training enables GUI understanding. Note: See paper for details.

Parameters to set

• gui_types — Role: Types of GUIs covered. How to set: Include web, mobile, desktop. Default/range: All types. Effect: More types improve generalization.
• resolution — Role: Screenshot resolution. How to set: High resolution for text readability. Default/range: 1120x1120. Effect: Higher resolution improves GUI text reading.

Validation checks

• The model should correctly identify GUI elements.
• Interaction predictions should be accurate.
• GUI text should be readable at the chosen resolution.

Failure modes

• GUIs change frequently, limiting data longevity.
• High resolution increases compute requirements.
• Some GUIs may have unusual layouts.

Adaptation notes for VLM training

• GUI agent data extends VLM capability to computer use.
• The approach applies to any visual interface understanding.
• Combine with general VLM data for balanced training.

Implementation notes

• Use high-resolution input for GUI text reading.
• Collect diverse GUI types for generalization.
• Evaluate on GUI benchmark tasks.

Evidence from the paper

• CogAgent creates GUI interaction data for VLM agent training.
• High resolution is critical for GUI understanding.
• The model navigates diverse GUIs through visual understanding.
• GUI agent capability extends VLM utility to computer use.

Source paper

• Title: CogAgent: A Visual Language Model for GUI Agents
• Year: 2024
• Venue: CVPR
• Paper ID: arxiv-2312.08914v2
• URL: http://arxiv.org/abs/2312.08914v2
• arXiv ID: 2312.08914v2