piperhex/remoteai/gui-automation

Name: gui-automation
Author: piperhex

Claude Code and agent skill by piperhex/remoteai

Trust tier 3

community

By piperhex
Design
piperhex/remoteai
packages/cua/vendor/libs/cua/skills/gui-automation/SKILL.md
Updated 22 Oct 2026
Version 1.0.01 version recorded

Source check pending

Checking that the SKILL.md file is still in the repository.

Use when you need to visually interact with a GUI: test buttons, fill forms, verify visual layouts, fuzz web pages, automate user flows, take screenshots, or perform end-to-end QA on any application. Works on cloud VMs, Docker containers, local machines, and sandboxes. Needs the `cua` CLI (curl -fsSL https://cua.ai/install.sh | sh).

Install

npx vskill@latest install piperhex/remoteai/gui-automation

SKILL.md

The same SKILL.md was published earlier as gabrielmoreira/agent-skills-mirror/gui-automation.

GUI Automation

CUA gives you eyes and hands on a real computer: see the screen, move the mouse, click, type, drag, and manage windows, like a human at the keyboard.

Use this skill for visual interaction that can't be done via shell or API.

Setup

bash

cua --version          # check install; if missing: curl -fsSL https://cua.ai/install.sh | sh
 
# Connect to target (pick one)
cua do switch my-sandbox                    # any sandbox from `cua sb ls`
cua do switch relay:<machine-id>            # a relay Space from `cua spaces ls`
cua do switch url http://127.0.0.1:3211 --as dev   # a cua-spacesd by URL
cua do-host-consent && cua do switch host   # local machine (one-time consent)

ANTHROPIC_API_KEY is optional. With it, cua do snapshot returns an
AI-annotated screen with element coordinates. Without it, use screenshot
and read the image yourself.

Workflow

Look → Act → Verify: repeat until done, then share:

bash

cua do screenshot          # look
cua do click 450 280       # act
cua do screenshot          # verify
cua trajectory view        # open the replay for the user

Re-screenshot after every UI change: coordinates go stale when the screen changes.

Scenarios

Click a button

bash

cua do screenshot
cua do click 450 280
cua do screenshot

Fill a form

bash

cua do screenshot
cua do click 400 200 && cua do type "Jane Doe"
cua do key tab            && cua do type "jane@example.com"
cua do key tab            && cua do type "SecureP@ss123"
cua do click 400 500
cua do screenshot

File upload dialog

bash

cua do click 350 400       # "Choose File"
cua do type "/home/user/report.pdf"
cua do key enter
cua do screenshot

Zoom in for precision clicks (host or small targets)

When clicking small or dense UI elements, especially on the host machine, zoom into the target window first. Coordinates become window-relative and screenshots show only that window, giving you higher effective resolution.

bash

cua do zoom "Google Chrome"   # crop to Chrome window; coords are now window-relative
cua do screenshot              # zoomed view, easier to locate small elements
cua do click 112 44            # precise click on a small tab or button
cua do screenshot              # verify
cua do unzoom                  # restore full-screen coords when done
cua do screenshot              # back to full desktop view

Use zoom any time click accuracy is uncertain. unzoom before switching
windows or when you need to see the full desktop again.

Drag and drop

bash

cua do window ls               # list open windows
cua do drag 150 300 650 400    # source → destination
cua do screenshot

Fuzz a form

bash

cua do screenshot
cua do click 400 200
cua do type "<script>alert(1)</script>"
cua do key tab && cua do type "'; DROP TABLE users; --"
cua do key tab && cua do type "AAAAAAAAAAAAAAAAAAAAAAA"
cua do click 400 500
cua do screenshot              # check for errors, crashes, unexpected behavior

Trajectory

Every action is auto-recorded to ~/.cua/trajectories/{machine}/{session}/.

bash

cua trajectory view            # open the replay viewer (do this at the end)
cua trajectory ls              # list sessions
cua trajectory clean           # delete sessions
cua do --no-record click 100 200   # disable recording for a single action

Tell the user where the trajectory is (cua trajectory ls prints the path).

Quick Reference

Action	Command
Connect to target	`cua do switch <sandbox>`
Screenshot	`cua do screenshot`
AI-annotated screen	`cua do snapshot ["instructions"]`
Click	`cua do click <x> <y> [left\	right\	middle]`
Double-click	`cua do dclick <x> <y>`
Type text	`cua do type "text"`
Press key	`cua do key <key>`
Hotkey	`cua do hotkey <combo>` (e.g. `ctrl+c`)
Scroll	`cua do scroll <direction> [amount]`
Drag	`cua do drag <x1> <y1> <x2> <y2>`
Move cursor	`cua do move <x> <y>`
Shell command	`cua do shell "command"`
Open URL/file	`cua do open <url\	path>`
List windows	`cua do window ls [app]`
Focus window	`cua do window focus <id>`
Zoom to window	`cua do zoom "App Name"`
Unzoom	`cua do unzoom`
View trajectory	`cua trajectory view`

Providers

Target	Example
a sandbox	`cua do switch my-sandbox` (see `cua sb ls`)
a relay Space	`cua do switch relay:<machine-id>` (`cua spaces ls`)
a spacesd URL	`cua do switch url http://host:3211 --as x`
this machine	`cua do switch host`

Legacy <provider> <name> pairs (cloud, docker, lume, ...) still work.

See references/command-reference.md for full argument syntax.

Security

Security scanPassed, 90 of 100Pattern checks on the skill's files.
Model reviewPassed

See the full security report

Quality evaluation

Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

See evaluation results

Source

SKILL.md in the repositoryCheckingpackages/cua/vendor/libs/cua/skills/gui-automation/SKILL.md

Popularity

Repository stars: 1,052
Repository forks: 41
7-day trend score: 0

Stars and forks are for the whole GitHub repository, not this skill alone. Not refreshed yet.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

Claude Code
Cursor
GitHub Copilot
Windsurf
Codex
Gemini CLI
Cline
Amp
Roo Code
Goose
Continue
Kilo Code
Junie
Trae
OpenHands
Qwen Code
AI Agent
Tabnine Chat
Sourcegraph Cody
Amazon Q
Firebase Genkit
Sweep
Devin
Phind
Mentat
Claude Desktop
Aider
GPT Pilot
Plandex
bolt.new
v0
Replit Agent
Anse
TypingMind
BetterChatGPT
LibreChat
Warp AI
Pieces
Fig AI

Embed badge

[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/piperhex/remoteai/gui-automation/badge)](https://verified-skill.com/skills/piperhex/remoteai/gui-automation)

Something wrong with this skill or its page? Tell us.