leehow/pipiui/cua-driver-operation

Cua Driver Operation

This private adaptation is pinned to Cua Driver 0.20.0. Use the exact pinned application/window target. desktop_open_application success already pins the exact window and brings it to the front: immediately call a fresh desktop_observe. Never use Command-Tab, Command-Shift-Tab, or the app switcher to locate or refocus that application. Observe before acting, prefer AX/semantic locators over pixels, batch coherent sequences, and obtain a fresh observation after consequential mutation. A timeout or driver error can mean unknown outcome: never repeat the mutation before observing.

Native macOS application menus are application-level AX surfaces and may be absent from the pinned content-window observation. Use one exact desktop_act action {type:"invoke_menu",path:[<top-level menu>,<immediate child>,...]}. Every path segment must be a literal live menu label; the Host keeps the call bound to the already-proven pinned PID and the driver fails closed on missing, ambiguous, disabled, or structurally mismatched segments. Never guess an outside-window coordinate for a menu bar or menu item.

Every state mutation consumes the current observation. Before any later click, scroll, key, typing, or other mutation, call desktop_observe with fresh:true and re-evaluate the returned UI. A mutation batch may contain only one state mutation; only a bounded wait may follow it in that same batch. fresh_observation_required_after_mutation is recoverable by observing, not by trying a different blind mutation.

Treat long-running target work as a temporal state, not as a failed Postcondition. A collapsible thinking, internal notes, side note, or similarly named disclosure label is not an error by itself. Never expand or open that disclosure to inspect raw chain-of-thought, and never count its presence as success. Only an explicit RPC, provider, runtime, or target error message is error evidence.

When the target shows an in-progress cluster — for example, its input is disabled and a stop control or progress status is visible — compare fresh observations instead of judging from one frame. A step, elapsed-time, token, or status value that advances is progress. Use one bounded wait of at most 15 seconds followed by a fresh observe, and repeat only while progress advances and the current worker deadline still permits it. This wait-and-observe loop must not extend or re-arm the worker deadline. Completion requires terminal UI evidence: the input is enabled, the stop control disappears, and a new reply or the declared Postcondition is visible. If an explicit error appears, or the progress evidence stops changing until the existing no-progress/deadline policy is exhausted, return the closed honest failure/block result; do not invent completion and do not wait indefinitely.

Every wait used for pending progress creates a mandatory post-wait observation fence. The Operator must not return any terminal outcome — completed, verified, blocked, or failed — until desktop_observe with fresh:true succeeds after that exact wait. A failed or timed-out observe does not clear the fence, and a pre-wait screenshot cannot be reused after waiting. Only the successful post-wait observation may judge terminal UI, Postconditions, explicit error evidence, or no-progress. Do not issue a wait unless the current worker deadline leaves time for its required fresh observe; never extend or re-arm that deadline. If cancellation or the deadline ends the worker before the required observation succeeds, let the runtime cancellation/deadline path close the worker instead of inferring a verdict from stale UI.

When a fresh observation exposes an interactive AXButton and a same-name container or static text, call desktop_locate with the exact interactive role and name first, then click only the returned binding/token. A container, group, or static text is evidence, never an interactive substitute. When a blocking onboarding, modal, or interstitial exposes an explicit non-destructive continue button consistent with the goal, resolve and activate that button first, then fresh-observe before continuing. Never blind-scroll or send hotkeys around a blocking layer.

When a fresh observation proves the pinned application is in the wrong target state and the task explicitly forbids every safe action needed to restore it (for example, clicking, closing, reopening, or recovery), return exactly {"outcome":"blocked","summary":"Target state requires manual restoration before retry.","recoveryDisposition":"manual_intervention","blockedReason":"target_state_mismatch","nextAction":"restore_target_state_manually"}. This is a closed manual-intervention signal, not ordinary failure prose. Do not use it for a failed locator, missing text with an allowed recovery path, a transient driver error, or an untried safe alternative.