arul28/ade/ade-computer-use

ADE computer use

This is the one entry point for computer use in ADE. Decide whether you need a screen at all, then pick the surface, then follow that surface's loop.

Detail files

Rarely needed detail lives next to this file. Read one when its topic comes up (or ade skill show ade-computer-use references/<file>):

Do you need a screen?

Go down this list and stop at the first step that answers the question:

1. No screen. A CLI, an API, a test, a log or the file the app writes.

Most "does it work" questions end here.

1. A surface that never opens a screen. A web page or localhost →

ade browser. An Electron app → ade app-control. An iOS app or an iOS SwiftUI view → ade apple. An Android app → ade android. They run in the background and never touch the user's screen, pointer or focus.

1. The lane's own screen (ade screen, on every host),

only for what steps 1 and 2 cannot reach:

• a native or custom-drawn Mac app: the Mac app you are building, a DAW

(FL Studio, Ableton, Logic), a game, a canvas or an OpenGL/Unity build;

• real Mac window behaviour: Open and Save panels, the menu bar, dragging

between windows or onto the Dock;

• Safari or WebKit, or an extension in a real browser, when the user

names it;

• an Electron app that will not start with a debug port, so App Control

cannot attach;

• proof or a recording of one of the above.

A lane screen costs the user something. It adds a display to their Mac, so macOS rearranges their screens while it is up. A permission or sign-in prompt can open on it, where the user cannot see it. So use it only for step 3, and stop it when you are done (see Rules that will bite you).

Pick the surface

App Control acts through an Electron app's DOM, so labels, selectors, test ids and hover work, and it records and proves the app's own window without a lane screen. An installed app (Slack, a packaged build of the app you work on) launches the same way as your dev build, with its binary and $ADE_APP_CONTROL_DEBUG_FLAGS (see ade-app-control). Only when the app refuses a debug port does Mac Desktop drive an Electron window by point.

ADE's browser is the default for web work. Attach to the user's own browser only when they ask for it, never on your own, and tell them which machine and tab you attached to.

Lane screens (ade screen) exist on macOS, Windows and Linux hosts. On Linux each lane gets its own X display, so nothing you do there touches anyone's screen; see Linux Desktop below.

Only ADE's surfaces stay off the user's screen. Anything else that clicks, types or captures — an MCP server, osascript or System Events, cliclick, screencapture, a browser-automation CLI, a Chrome you open yourself — acts on the user's real screen, pointer and focus. Do not use them for task work. When an ADE surface cannot do a step, say which step and why, and ask; do not switch tools silently.

Note: ade desktop is a different command. It launches the ADE desktop app.

A device on another machine

ade apple, ade screen and ade app-control take --machine <name> to drive that machine's simulator, screen or app. The person at that Mac sees your session and can end it; stop it yourself when done. Flags, where proof lands, and building there: references/other-machines.md.

The same answer on every surface

Every surface works the same way: observe, act on a handle from the last observation, then check the result. Every acting command (click, type, press, fill, scroll, drag) reports two lines:

• hit: — the element the command actually hit.
• effect: — observed (ADE saw the screen change), unconfirmed (ADE

looked and saw no change), not_checked (ADE did not look), or, in the browser, waiting (the action started a navigation that waits for the user to allow this chat to use the ADE browser; observe again after they answer).

When the effect is unconfirmed, observe again before you act again or report the step. Do not repeat the action blindly.

An unconfirmed answer can carry a third line, next:, naming the one input method to try next (observe, background_input, real_input, lease, browser) and often a command to run. Follow next: instead of a guess. Today only the Mac Desktop prints it. What each value means: references/mac-screen-details.md.

Proof per surface

Stage before showcase proof: close or minimize every window that is not the subject (ade screen windows --text). Per surface, and the steps, in references/proof-staging-and-demos.md and ade-proof-artifacts.

ade proof capture and ade proof record capture the lane's display. They refuse the user's real screen unless you pass --real-screen. Pass it only when the user asks for proof of their own screen. The ade-proof-artifacts skill covers attaching files and confirming a filing.

Each of these commands prints a cite: line, such as ![Settings saved](ade-proof://<id>). Paste it into your answer directly under the claim it proves: the picture shows there, and a video plays there. For a before/after, use a proof-compare block (see ade-proof-artifacts). Caption each item, and say what it does not show.

Show the user

ade ui show mac-desktop     # also: apple, browser, proof, app-control
ade screen show --text
ade screen show --floating --text
ade app-control show --floating --text

The lane's own screen

Mac Desktop

Each lane owns a private macOS screen. ade screen ... creates it, parks apps on it, and drives them through the Accessibility API. ADE captures the user's real display only through --real-screen, and drives their pointer only under a lease, so background and Accessibility clicks need no ask. The one exception is real input (--real), which asks for that lease below. While a lane screen is up, macOS may rearrange the user's screens around it for this session.

For web tasks, use ade browser (step 2 above). Open Safari or another browser here only when the user names that app or asks for the Mac Desktop.

On Windows and Linux the same ade screen verbs drive that host's screen; read the short section for your host below first. --lane defaults to ADE_LANE_ID. In a chat, the command is pinned to that chat's lane: --lane naming a different lane is refused, not silently swapped.

Windows Desktop

On a Windows host, ade screen is Windows Desktop, with the Mac verbs below. Start every task with ade screen status --text and follow its next line. The default private seat is a separate session of the user's account, not their screen. The shared seat is the user's main desktop and needs their yes (ade screen start --shared --reason "<what for>" --text); never pass --consent or --allow-prompt, and never use --role cto. Setup, a saved password, a locked PC or an administrator window need the user: ask, never do it yourself, and never ask for the password in chat. Editors such as Notepad reopen the user's files, which can hold secrets: open a blank file for the task. Seats, the ask-the-user table, Windows specifics and proof: references/windows-desktop.md.

Linux Desktop

On a Linux host the lane screen is the lane's own X display, so point clicks, drags, keys and --real need no lease. Missing packages (Xvfb, AT-SPI, ffmpeg): you cannot install them; ask the user. Browsers, Electron apps, stalled apps, modifier keys and windows: references/linux-desktop.md.

Check each step before you report it

A click, type or press that returns ok only means ADE sent the input. It does not mean the app did what you wanted. After each step that matters, confirm it: read the screen the command printed, ade screen wait --label "<text>", ade screen observe, or a screenshot. Report only what you confirmed. If a step failed, say which step, and do not describe the result you meant to get. Before record stop, confirm the final state, so the video ends on it.

Common tasks

Use these directly; you do not need --help for them.

Operating loop