fnosp/fnos-dsh/computer-automation

Desktop Computer Automation

CRITICAL RULES — VIOLATIONS WILL BREAK THE WORKFLOW:

1. Never run midscene commands in the background. Each command must run synchronously so you can read its output (especially screenshots) before deciding the next action. Background execution breaks the screenshot-analyze-act loop.

2. Run only one midscene command at a time. Wait for the previous command to finish, read the screenshot, then decide the next action. Never chain multiple commands together.

3. Allow enough time for each command to complete. Midscene commands involve AI inference and screen interaction, which can take longer than typical shell commands. A typical command needs about 1 minute; complex act commands may need even longer.

4. Always report task results before finishing. After completing the automation task, you MUST proactively summarize the results to the user — including key data found, actions completed, screenshots taken, and any relevant findings. Never silently end after the last automation step; the user expects a complete response in a single interaction.

5. Only minimize windows, never close them unless explicitly asked. When you need to dismiss or get a window out of the way, minimize it instead of closing it. Do not close any app or window unless the user explicitly asks you to do so.

Control your desktop (macOS, Windows, Linux) using npx -y @midscene/computer@1. Each CLI command maps directly to an MCP tool — you (the AI agent) act as the brain, deciding which actions to take based on screenshots.

What act Can Do

Inside a single act call on desktop, Midscene can move the mouse, click, double-click, right-click, drag items, type or clear text, scroll, press single keys or keyboard shortcuts, and work through multi-step interactions on whatever is visible on the selected display.

Prerequisites

Midscene requires models with strong visual grounding capabilities. The following environment variables must be configured — either as system environment variables or in a .env file in the current working directory (Midscene loads .env automatically):

MIDSCENE_MODEL_API_KEY="your-api-key"
MIDSCENE_MODEL_NAME="model-name"
MIDSCENE_MODEL_BASE_URL="https://..."
MIDSCENE_MODEL_FAMILY="family-identifier"

Example: Gemini (Gemini-3-Flash)

MIDSCENE_MODEL_API_KEY="your-google-api-key"
MIDSCENE_MODEL_NAME="gemini-3-flash"
MIDSCENE_MODEL_BASE_URL="https://generativelanguage.googleapis.com/v1beta/openai/"
MIDSCENE_MODEL_FAMILY="gemini"

Example: Qwen 3.5

MIDSCENE_MODEL_API_KEY="your-aliyun-api-key"
MIDSCENE_MODEL_NAME="qwen3.5-plus"
MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
MIDSCENE_MODEL_FAMILY="qwen3.5"
MIDSCENE_MODEL_REASONING_ENABLED="false"
# If using OpenRouter, set:
# MIDSCENE_MODEL_API_KEY="your-openrouter-api-key"
# MIDSCENE_MODEL_NAME="qwen/qwen3.5-plus"
# MIDSCENE_MODEL_BASE_URL="https://openrouter.ai/api/v1"

Example: Doubao Seed 2.0 Lite

MIDSCENE_MODEL_API_KEY="your-doubao-api-key"
MIDSCENE_MODEL_NAME="doubao-seed-2-0-lite"
MIDSCENE_MODEL_BASE_URL="https://ark.cn-beijing.volces.com/api/v3"
MIDSCENE_MODEL_FAMILY="doubao-seed"

Commonly used models: Doubao Seed 2.0 Lite, Qwen 3.5, Zhipu GLM-4.6V, Gemini-3-Pro, Gemini-3-Flash.

If the model is not configured, ask the user to set it up. See Model Configuration for supported providers.

Commands

Connect to Desktop

npx -y @midscene/computer@1 connect
npx -y @midscene/computer@1 connect --displayId <id>

Connect via RDP

Use RDP mode to drive a remote Windows desktop instead of the local machine. Providing --host switches connect to RDP and routes every subsequent command (act, tap, take_screenshot, assert, disconnect) through the RDP helper binary bundled with @midscene/computer. The local mouse/keyboard is not touched.

Minimum example:

npx -y @midscene/computer@1 connect \
  --host rdp.example.com \
  --username Administrator \
  --password "$RDP_PASSWORD"

All RDP options for connect (RDP mode is activated when --host is set; the other flags are optional):

• --host <fqdn-or-ip> — RDP host. Required to enter RDP mode.
• --port <number> — RDP port (default 3389).
• --username <user> — RDP user account.
• --password <secret> — RDP password. Prefer reading from an environment variable, secrets manager, or interactive prompt; never paste it into a shared transcript.
• --domain <domain> — Active Directory / NTLM domain.
• --security-protocol <auto|tls|nla|rdp> — Security protocol negotiation. Defaults to auto.
• --ignore-certificate — Skip TLS certificate validation. Use only for trusted dev hosts with self-signed certs.
• --admin-session — Attach to the admin/console session (equivalent to mstsc /admin).
• --desktop-width <px> and --desktop-height <px> — Request a specific remote desktop resolution. The actual size is whatever the RDP server negotiates back; e.g. requesting 1024x768 against a host that pins 1280x720 will land on 1280x720. Confirm the negotiated size with listdisplays --host ... --username ... --password ... after connect.

Notes specific to RDP mode:

• --displayId and --headless are ignored in RDP mode. A connected RDP session always exposes a single virtual display whose size is whatever the server negotiated.
• Two display-listing commands exist and behave differently — pick the right one:
• list_displays (with underscore, platform tool) — enumerates local physical displays only. Does not accept RDP flags. Useless after an RDP connect.
• listdisplays (no underscore, action tool) — accepts the same RDP flags as connect/take_screenshot/etc. In RDP mode it returns the negotiated virtual display, e.g. [{ "id": "...", "name": "RDP 10.70.86.26:3389 (1280x720)", "primary": true }]. Use this to verify the actual resolution.
• The RDP transport uses a native helper binary shipped inside @midscene/computer. If you see RDP helper binary not found errors, the optional bin/<platform>/rdp-helper was stripped from your install — reinstall the package or unpack a fresh tarball.
• Treat RDP credentials as secrets: do not commit .env files containing --password to the repo; prefer export RDP_PASSWORD=... in the current shell and reference it as --password "$RDP_PASSWORD".
• Latency expectations: every CLI invocation is a fresh node process, so each command re-establishes the RDP session. Budget roughly:
• connect / take_screenshot / keyboardpress / scroll: ~5 s (node startup + RDP TLS+NLA handshake + first frame).
• act / assert / tap --locate: ~5 s + AI inference + any planned sub-actions; expect 8–20 s end-to-end for typical interactions.
• The RDP handshake itself is ~700 ms; the rest is unavoidable cold-start cost in the CLI shape.
• Connect failure diagnostics: when connect fails, the first line of stderr is the actionable error (e.g. connect_failed: Failed to connect to RDP server: ERRCONNECT_LOGON_FAILURE: Logon failed.). The subsequent stack trace is diagnostic noise — read the first line, then check credentials/network. Common ERRCONNECT_* causes:
• LOGON_FAILURE — bad username/password/domain.
• CONNECT_TRANSPORT_FAILED — host unreachable or RDP port blocked. Verify with nc -zv <host> 3389.
• TLS_CONNECT_FAILED — TLS handshake rejected. Try --ignore-certificate for self-signed dev hosts, or pin --security-protocol nla.

After connect --host ... succeeds, the rest of the workflow (act, tap --locate, assert, take_screenshot, listdisplays, report-tool, disconnect) is identical to local mode — just remember to pass the same --host/--username/--password/--ignore-certificate flags to every subsequent command, since each CLI invocation is stateless and reconnects.

List Displays

npx -y @midscene/computer@1 list_displays

Take Screenshot

npx -y @midscene/computer@1 take_screenshot

After taking a screenshot, read the saved image file to understand the current screen state before deciding the next action.

Perform Action

Use act to interact with the computer and get the result. It autonomously handles all UI interactions internally — clicking, typing, scrolling, waiting, and navigating — so you should give it complex, high-level tasks as a whole rather than breaking them into small steps. Describe what you want to do and the desired effect in natural language:

# specific instructions
npx -y @midscene/computer@1 act --prompt "type hello world in the search field and press Enter"
npx -y @midscene/computer@1 act --prompt "drag the file icon to the Trash"
 
# or target-driven instructions
npx -y @midscene/computer@1 act --prompt "search for the weather in Shanghai using the Chrome browser, tell me the result"

Assert Current Screen State

Use assert to verify that the current screen satisfies a natural language condition. It does not perform UI actions; it checks the visible screen state and passes only when the assertion is true. Use this for validation, QA checks, and final state verification after act.

npx -y @midscene/computer@1 assert --prompt "there is a login button visible"
npx -y @midscene/computer@1 assert --prompt "the active window shows a saved confirmation message"
npx -y @midscene/computer@1 assert --displayId 1 --prompt "the file picker is open"

By default a failed assertion throws an AI-generated reason. Pass --message to throw a custom error message instead, which is useful for surfacing the intended outcome in QA and CI logs.

npx -y @midscene/computer@1 assert \
  --prompt "the export completed dialog is visible" \
  --message "the export should finish after clicking Save"

When the assertion needs to compare against a reference image (icon, logo, screenshot), pass --image for the URL/path and --image-name for its display name. Each --image may be an http(s) link, a data: URI, or a local file path. Repeat both flags in matching order when you need to attach more than one image. Add --convertHttpImage2Base64 true when the model cannot reach the URL directly. Requires @midscene/computer@1.9.0+.

npx -y @midscene/computer@1 assert \
  --prompt "the active window matches the supplied reference screenshot" \
  --image "https://example.com/reference.png" \
  --image-name "reference" \
  --convertHttpImage2Base64 true
 
# or with a local file
npx -y @midscene/computer@1 assert \
  --prompt "the visible icon matches the supplied logo" \
  --image "./fixtures/logo.png" \
  --image-name "logo"
 
# multiple reference images — pair --image and --image-name by order
npx -y @midscene/computer@1 assert \
  --prompt "the active window matches both the icon and the logo" \
  --image "./fixtures/icon.png" --image-name "icon" \
  --image "./fixtures/logo.png" --image-name "logo"

Record and Assert Transient UI

Use a recording when the state to verify may disappear before a current-screen assertion runs, such as a toast, loading banner, animation, or transition:

# Terminal 1: keep this foreground command running
npx -y @midscene/computer@1 record start --displayId 1 \
  --output ./save-observation.json
 
# Terminal 2, while Terminal 1 records
npx -y @midscene/computer@1 act --displayId 1 \
  --prompt "click the Save button"
 
# Send Ctrl+C to Terminal 1 and wait for the saved-path message, then assert
npx -y @midscene/computer@1 assert --displayId 1 \
  --record ./save-observation.json \
  --prompt "a saved confirmation appeared"