buildoak/fieldwork-skills/peekaboo

Peekaboo

Native macOS GUI automation via peekaboo, using an AX-first hybrid approach that prioritizes accessibility-tree interaction when reliable and switches to screenshot, coordinate, JavaScript, or vision-based fallbacks when AX coverage breaks down in real apps.

Terminology used in this file:

• AX (Accessibility tree): The macOS accessibility hierarchy used for deterministic element discovery and interaction (B1, T2, etc.).
• peekaboo: The macOS automation CLI used for discovery (see, image), interaction (click, type, hotkey), and app/window/dialog control.
• UI-TARS: Vision model server used when AX discovery fails or times out on complex UIs.
• FSM: Finite State Machine controlling retries, transitions, and recovery paths.
• Retina scale: Display pixel ratio (typically 2.0) used to map screenshot coordinates correctly on Retina displays.
• snapshot ID: The snapshot_id returned by see; required for stable element resolution and invalidated by UI changes.

Setup

brew install jq
brew install --cask peekaboo
mkdir -p ./scripts
chmod +x ./scripts/peekaboo-safe.sh ./scripts/health-check.sh
./scripts/health-check.sh

• Grant Accessibility and Screen Recording permissions for your terminal agent host (for example, Terminal, iTerm2, Claude Code).
• Use ./scripts/peekaboo-safe.sh as the only invocation entrypoint for peekaboo commands.
• For full setup details and troubleshooting, see references/installation-guide.md if present in your local skill copy.

Staying Updated

This skill ships with an UPDATES.md changelog and UPDATE-GUIDE.md for your AI agent.

After installing, tell your agent: "Check UPDATES.md in the peekaboo skill for any new features or changes."

When updating, tell your agent: "Read UPDATE-GUIDE.md and apply the latest changes from UPDATES.md."

Follow UPDATE-GUIDE.md so customized local files are diffed before any overwrite.

Quick Start

Run a 3-step baseline flow: health check, discover, interact.

# 1) Health check
./scripts/health-check.sh
 
# 2) Discover UI state
./scripts/peekaboo-safe.sh see --app Safari --json --path /tmp/peekaboo/
 
# 3) Interact
./scripts/peekaboo-safe.sh click --on B1 --app Safari

Hard Rules (MUST FOLLOW)

1. All peekaboo calls MUST use `./scripts/peekaboo-safe.sh` wrapper - never direct peekaboo calls
2. Sequential execution only - no parallel GUI operations (causes race conditions)
3. NEVER run `tccutil reset` - resets ALL app permissions, not just one
4. Always `--mode screen` - window mode returns 698B stubs (beta3 bug)
5. Health check required - run ./scripts/health-check.sh before automation
6. Fresh snapshots mandatory - re-run see before each click sequence (IDs expire)
7. Use see --analyze for Q&A - 50x more token-efficient than full screenshots

Navigation Strategy: Navigate Programmatically, Interact Visually

Core principle: Never waste turns clicking through UI to reach a target. Get there programmatically, then use peekaboo for the last-mile interaction.

Hostile AX apps (use URL scheme / AppleScript only):

• System Settings - SwiftUI toggles invisible to AX roles, sidebar clicks misfire

Routing Algorithm (Interaction Tier)

if target_is_native_macos_ui():
    use_ax_interaction()  # click --on B1, type, hotkey
elif target_is_swiftui_toggle():
    use_screenshot_coordinate_click()  # screenshot -> identify -> click --coords
elif target_is_web_form():
    use_safari_js_injection()  # osascript JavaScript
elif target_is_web_button():
    use_coordinate_click()  # click --coords X,Y --no-auto-focus
elif see_timeout_complex_app():
    start_ui_tars_server()  # vision provider required
elif dialog_or_sheet():
    add_no_auto_focus_flag()  # prevent focus conflicts
else:
    escalate_to_human()

FSM State Machine

Bug-Specific Transitions

• see timeout -> start UI-TARS -> retry DISCOVER
• Safari AX click misfires -> force coordinate click path
• type targets URL bar -> force JS injection path
• daemon file-not-found -> daemon run --mode manual &
• Chrome window capture fail -> switch to Safari
• System Settings AX sidebar misfire -> use URL scheme: open "x-apple.systempreferences:com.apple.PANE_ID"
• SwiftUI toggle invisible to AX -> screenshot + coordinate click (not AX click)

Core Commands (Canonical Forms Only)

Discovery

• ./scripts/peekaboo-safe.sh see --app AppName --json --path /tmp/peekaboo/
• ./scripts/peekaboo-safe.sh see --analyze "What is the current state?" --app AppName
• ./scripts/peekaboo-safe.sh image --mode screen --path /tmp/peekaboo/screenshot.png

Interaction

• ./scripts/peekaboo-safe.sh click --on B1 --snapshot "$sid" --app AppName
• ./scripts/peekaboo-safe.sh click --coords X,Y --no-auto-focus --app AppName
• ./scripts/peekaboo-safe.sh type "text" --return --profile human --app AppName
• ./scripts/peekaboo-safe.sh hotkey --keys "cmd,w" --app AppName

App Control

• ./scripts/peekaboo-safe.sh app switch --to AppName
• ./scripts/peekaboo-safe.sh window focus --window-title "Title" --app AppName
• ./scripts/peekaboo-safe.sh menu click --path "File > Save As..." --app AppName

Safari Workarounds

• Navigate: ./scripts/peekaboo-safe.sh open "https://url" --app Safari
• Form fill: osascript -e 'tell application "Safari" to do JavaScript "document.querySelector(\"input[name=field]\").value = \"value\"" in document 1'
• Button click: ./scripts/peekaboo-safe.sh click --coords X,Y --no-auto-focus --app Safari

Agent Mode

• ./scripts/peekaboo-safe.sh agent "Natural language task" --model gpt-4o --max-steps 10
• ./scripts/peekaboo-safe.sh agent --resume --max-steps 5

Token Budget

see --analyze ~100 tokens | see --json ~600 | image --mode screen ~1400. Soft limit 50 turns, hard limit 100. Prefer see --analyze over full AX trees. Details in references/token-budget.md.

Capability Matrix

Playbook Index

Each playbook is a self-contained field-tested recipe. Status: validated = tested in production, experimental = untested or partially tested.

Reference Index

• references/commands.md - Complete command reference with all flags
• references/safari-workarounds.md - Web automation canonical patterns
• references/patterns.md - Common automation workflows and recipes
• references/failures.md - Error recovery procedures and symptom table
• references/validated-tests.md - Field validation results and capabilities
• references/token-budget.md - Token costs and optimization strategies
• references/hard-tests.md - Edge-case and stress test catalog
• references/installation-guide.md - Full setup from zero to working

Pre-flight Checklist

./scripts/health-check.sh
# Must return: cli_exists=true, accessibility_granted=true, ax_test_success=true
mkdir -p /tmp/peekaboo

Error Recovery

After 3 retries on same state: log state + evidence screenshot + escalate to HUMAN_HANDOFF. Quick recovery: app switch + fresh see + health-check.sh.