yliust/tactile/tactile-windows

Tactile Windows

Use this skill to inspect and operate local Windows applications through UI Automation, input events, targeted OCR, and bounded observe-plan-act workflows.

Requirements

• Run on Windows with a working Python 3.11+.
• The Windows UI Automation runtime is bundled at vendor/WindowsUseSDK; no separate WindowsUseSDK checkout is required for a normal clone.
• Pass --repo <path> or set WINDOWS_USE_SDK_ROOT only when intentionally testing a different WindowsUseSDK checkout.
• For LLM-driven workflows, install the optional Python dependencies from pyproject.toml and set TACTILE_OPENAI_API_KEY; optionally set TACTILE_OPENAI_BASE_URL and TACTILE_MODEL.

Primary CLI

Use the skill-local CLI from this directory:

.\bin\tactile-windows.cmd <command> ...

PowerShell can also call:

powershell -ExecutionPolicy Bypass -File .\bin\tactile-windows.ps1 <command> ...

The CLI wraps scripts/windows_interface.py, which uses the bundled vendor/WindowsUseSDK/WindowsUseSDK.ps1 by default. This skill keeps Tactile-specific guidance, app guides, and session-scoped artifacts next to the Windows command surface.

App Resolution Rule

When the user asks to operate an app, resolve the target from installed apps or current top-level windows before acting. Do not rely only on a display name, because window titles, localized names, executable names, app IDs, and multiple windows can differ.

Prefer compact discovery and keep the returned hwnd, frame, title, and process information for subsequent commands:

.\bin\tactile-windows.cmd list-apps --query Calculator
.\bin\tactile-windows.cmd open Calculator
.\bin\tactile-windows.cmd observe Calculator

For multi-window tasks, re-list or re-observe after a dialog, popup, child process, login window, media viewer, or secondary app window appears.

Core Workflow

1. Resolve and activate the target app/window with list-apps, open, or observe.
2. Start every multi-step task with structured observation: elements --target <app> or observe <app>.
3. For sparse Electron/WebView apps, run elements --view raw before OCR. Raw UIA paths can still be used by uia and ocr.
4. Use the observation priority UIA > targeted OCR/probe > workflow visual context > raw coordinates. Do not use screenshots as the primary reasoning path. Visual inspection of screenshots or OCR capture images is the last resort: use it only after UIA, focused probes, and targeted OCR cannot answer the question, keep the crop tight, and name why the fallback was needed.
5. Treat every click, keypress, text input, scroll, and wait as a single-action boundary. Re-observe before deciding the next action.
6. For high-risk external actions such as sending messages, posting comments, payments, destructive edits, account changes, and form submissions, split the task into locate, draft/open controls, verify, submit.
7. Before final submit/send, verify the active window, target recipient/context, and draft/action state from the latest observation or targeted OCR.
8. If permissions or platform limits block automation, report the specific Windows/UIA/OCR limitation and what was verified instead.

CLI Examples

$artifactDir = .\bin\tactile-windows.cmd artifact-dir
.\bin\tactile-windows.cmd list-apps --query Feishu
.\bin\tactile-windows.cmd open Feishu --output "$artifactDir\feishu-open.json"
.\bin\tactile-windows.cmd elements --target Feishu --view control --output "$artifactDir\feishu-elements.json"
.\bin\tactile-windows.cmd elements --target Feishu --view raw --output "$artifactDir\feishu-raw.json"
.\bin\tactile-windows.cmd probe --target Feishu --x 138 --y 35
.\bin\tactile-windows.cmd uia click 123456 "uia:123456:root.children[0]"
.\bin\tactile-windows.cmd input keypress ctrl+f --hwnd 123456
.\bin\tactile-windows.cmd input streamtext "hello" --hwnd 123456
.\bin\tactile-windows.cmd ocr --hwnd 123456 --rect 100,200,360,80
.\bin\tactile-windows.cmd wechat-send-message --chat "<contact>" --message "<message>"
.\bin\tactile-windows.cmd workflow "open Calculator and calculate 2+3" -- --target Calculator --execute --debug-observation
.\bin\tactile-windows.cmd plan-log "$artifactDir\workflow-run.json"

Fast App Paths

• Use wechat-send-message for routine WeChat contact/group messaging only after the current WeChat session has a verified search-focus path, or when the intended chat is already open and the flow can skip searching. The helper resolves WeChat once, computes frame-relative profile regions, selects any existing compose draft before pasting, uses direct Win32 input by default, focuses the compose area, presses Enter to send, and writes one JSON result. For routine short messages, prefer --require-title-match --no-draft-ocr so the command performs one targeted recipient check and avoids the slower, less reliable compose OCR pass.
• For a routine WeChat send, do not spend time on broad discovery once open 微信 returns a visible main window with a valid hwnd and frame. list-apps may surface minimized/offscreen auxiliary WeChat windows such as media viewers; ignore those after open recovers the main chat window. Skip dry-run, extra artifact-dir, and repeated observe/elements passes unless the main window, focus path, or recipient state is ambiguous.
• Do not split WeChat sends into wechat-send-message --draft-only followed by a manual input keypress enter. The helper's contact-search Enter is not a harmless navigation action if focus missed the search box; it can send the contact name into the currently open chat, and a later title check can still pass when that chat is already the target. If the user asks to draft only, stop after the draft and report that it is prepared. If the user asks to send, use one verified send path rather than a helper draft plus a separate submit.
• Before typing a WeChat contact name into search, check whether the intended chat is already open. If targeted title verification already matches the recipient, skip search entirely and operate only on the compose area. Never paste the recipient/search token into the UI unless the latest UIA observation or targeted probe confirms the search field is focused; in sparse WeChat UIA, a click on the profile search center followed by the search placeholder disappearing/clearing in a targeted probe is enough focus evidence for a routine helper send.
• For WeChat send verification, prefer structured state and targeted UIA/probe checks over broad visual fallback. Screenshots/OCR are supporting evidence only; if focus, active chat, or draft state is uncertain, stop or re-locate through UIA/profile regions instead of pressing Enter.
• For WeChat read-and-reply tasks, treat message reading as content extraction rather than send verification. First verify the active chat title, then use the latest frame-relative message-region OCR crops. If the OCR text is unreadable, retry with a tighter line-specific crop or a UTF-8 readback of the saved OCR JSON before opening an image. Inspect the OCR capture image only when the reply depends on exact content and text extraction is still semantically ambiguous.
• For important WeChat messages that contain meeting links, meeting numbers, or other must-preserve details, prefer a single-line message or send short sequential messages. Some shells and chat controls can treat multiline CLI arguments as separate input; verify the final visible chat content, not only command success.
• Do not add a separate dry-run, manual character-code check, or extra OCR pass for routine user-supplied Chinese text. Console rendering may show mojibake even when JSON and Python strings are correct; treat status: success plus title_verification.matched: true as the result signal, and only investigate encoding if the JSON bytes/value are wrong when read as UTF-8.
• Distinguish console mojibake from OCR corruption. If Get-Content or PowerShell output shows Chinese as mojibake, re-read the saved JSON with an explicit UTF-8 path (Get-Content -Encoding UTF8 or another UTF-8 byte reader) before assuming the action or OCR failed. If the UTF-8 JSON itself contains mojibake-like OCR text while the crop image is readable, treat that OCR text as corrupted for exact message reading and do not base a semantic reply on it.
• Use --draft-only only when the user asks to prepare without sending or the message contains high-value details that need human-visible review. Use --dry-run to inspect computed regions without typing, --sdk-input when direct Win32 input is blocked, --keep-existing-draft only when intentional, --send-method button only when Enter is known not to send in that WeChat setup, and --require-draft-match only when compose OCR reliably returns readable text for the current language.
• For unsupported WeChat flows such as Moments, comments, media viewers, or unusual popups, fall back to the manual observe-plan-act workflow in references/app-guides/WeChat.md.
• For Tencent Meeting scheduling, read references/app-guides/Tencent-Meeting.md. Treat schedule, time-picker, and invitation dialogs as separate windows; after invoking 预定会议, re-run list-apps --query wemeetapp before retrying clicks in the main window.
• Use feishu-switch-org before Feishu/Lark messaging when the user names an account or organization, then use feishu-send-message --draft-only for risky or first-time sends. Feishu search must choose a verified contact/result title row from OCR, not a row where the target only appears in 包含..., group-message metadata, search history, or snippets. feishu-send-message automatically retries Enter against the compose-area Chrome_RenderWidgetHostHWND child when the top-level Feishu hwnd leaves the draft visible. If doing the fallback manually, do not reopen the chat or paste again; use the child native_window_handle, click the verified compose area, send input keypress enter --hwnd <child-hwnd>, and verify the compose placeholder or cleared draft. See references/app-guides/Feishu-Lark.md for the detailed fallback.

UIA, OCR, And Coordinates

• Prefer uia click <hwnd> <uia_path> for click-like actions. It resolves the element and clicks the current center from UIA data.
• Prefer uia focus, uia select, or uia invoke only when that semantic action is the goal or known to work for the app.
• Use probe when one candidate control is ambiguous. It combines UIA FromPoint with a local OCR crop.
• Use OCR only after UIA is insufficient. Prefer targeted OCR with --hwnd + --uia-path, --hwnd + --rect, or probe; whole-window OCR is a fallback.
• Use input click only from a fresh UIA element center, OCR result center, app-guide profile region, or same-window frame-relative calculation.
• Do not reuse absolute coordinates across app activation, restore, monitor move, DPI scaling change, popup creation, or window resize.
• If a UIA action returns success but the visible state does not change, treat success as inconclusive. Re-observe, then use one current center-coordinate fallback only if the action is low risk.

Text Input

• Prefer input streamtext or input writetext for normal typing. They use real input events and do not borrow the clipboard.
• Use input pastetext for long text, multiline content, WeChat Chinese input, or when Unicode streaming fails.
• After text entry into search, compose, or form fields, re-observe or use targeted OCR to require the expected text or resulting state before submit.
• If the current observation already shows the intended text in the correct target field, do not write it again. Submit, finish, or clear and replace only when the visible text is wrong.

Workflow Mode

Use workflow for multi-step work. The bundled WindowsUseSDK includes workflows/llm_app_workflow.py: