dingxingdi/paper_fast_search_backup/cross-modal-alignment-and-dialogue-grounding

Skill: cross-modal-alignment-and-dialogue-grounding

1. Capability Definition & Real Case

• Professional Definition: The capability to temporally align visual events with speech, sound, subtitles, or speaker identity, so that the answer depends on synchronized multimodal evidence rather than on vision-only or audio-only cues.
• Dimension Hierarchy: Multimodal and Generative Interpretation->Cross-Modal Grounding->cross-modal-alignment-and-dialogue-grounding

Real Case

[Case 1]

• Initial Environment: A long kitchen-and-living-room family video with clear speech, object interactions, and repeated refrigerator visits. The needed clue spans one spoken instruction and a later visual storage action.
• Real Question: Which shelf in the refrigerator, counting from the top, does Cary's family usually put the wine they bought on?
• Real Trajectory: Locate the spoken instruction about where yogurt should go, normalize 'second floor' as the second shelf from the top, find the later refrigerator placement of the wine, and infer the wine shelf by aligning the later visual placement with the earlier verbal reference.
• Real Answer: Third.
• Why this demonstrates the capability: The answer emerges only after synchronizing dialogue content with later visual evidence. This tests whether the agent can align two modalities over time instead of treating speech and video as unrelated streams.

Pipeline Execution Instructions

To synthesize data for this capability, you must strictly follow a 3-phase pipeline. Do not hallucinate steps. Read the corresponding reference file for each phase sequentially:

1. Phase 1: Environment Exploration

Read the exploration guidelines to discover raw knowledge seeds: references/EXPLORATION.md

1. Phase 2: Trajectory Selection

Once Phase 1 is complete, read the selection criteria to evaluate the trajectory: references/SELECTION.md

1. Phase 3: Data Synthesis

Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data: references/SYNTHESIS.md