dingxingdi/paper_fast_search_backup/fine-grained-widget-grounding

Skill: fine-grained-widget-grounding

1. Capability Definition & Real Case

• Professional Definition: The capability to localize and act on a precise GUI widget in dense, repetitive, or small-scale layouts, especially when the target belongs to a long-tail element type such as icons, toggles, radio buttons, or small action affordances. The agent must separate the true target from visually similar distractors and execute a pixel-accurate action on the right element.
• Dimension Hierarchy: GUI Perception and Environment Modeling->Element Grounding->fine-grained-widget-grounding

Real Case

[Case 1]

• Initial Environment: An email client shows an inbox with several messages arranged in a vertical list. Each row contains multiple compact icons, and the target action is the small reply icon associated specifically with the second email rather than the first or third.
• Real Question: Click the reply icon near the second email.
• Real Trajectory: Count the email rows from top to bottom, inspect the action cluster attached to the second row, distinguish the reply icon from adjacent icons such as archive or more-actions, and click that precise icon.
• Real Answer: The reply composer for the second email is opened.
• Why this demonstrates the capability: This case stresses fine-grained localization because multiple nearly identical icons are present and the instruction depends on ordinal position. The agent must jointly solve row counting, icon classification, and precise pointing under a dense layout. That makes the task a direct probe of high-resolution widget grounding rather than generic page comprehension.

Pipeline Execution Instructions

To synthesize data for this capability, you must strictly follow a 3-phase pipeline. Do not hallucinate steps. Read the corresponding reference file for each phase sequentially:

1. Phase 1: Environment Exploration

Read the exploration guidelines to discover raw knowledge seeds: references/EXPLORATION.md

1. Phase 2: Trajectory Selection

Once Phase 1 is complete, read the selection criteria to evaluate the trajectory: references/SELECTION.md

1. Phase 3: Data Synthesis

Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data: references/SYNTHESIS.md