desplega-ai/agent-swarm/agent-browser

agent-browser

agent-browser (vercel-labs) is the swarm's browser-automation CLI. It drives a headless Chromium over CDP and returns accessibility-tree snapshots with compact @eN refs. Use it for any browser task: verify a UI change, take a screenshot, walk a page, fill a form, extract page text, reproduce a frontend bug, or attach UI evidence to a PR, issue, Linear comment, or Slack thread.

Do not use qa-use. It is not installed in the worker image any more.

Step 0: check that the CLI is present

The full worker image (agent-swarm-worker:latest) ships agent-browser and a headless Chromium. The slim image (:slim) has neither. The first command of any browser work is:

command -v agent-browser

If it prints nothing, you are on the slim image. Say so in your progress report ("agent-browser is not available on this worker image") and fall back to manual QA: write the exact steps a human should run and mark the browser test cases as Blocked, not failed. Do not install a browser or the CLI yourself.

Step 1: load the version-matched guide

The CLI bundles its own usage guide. Load it once per task before the first browser command. Do not guess flags from memory.

agent-browser skills get core          # overview + common patterns
agent-browser skills get core --full   # adds the full command reference
agent-browser skills list              # specialized guides (dogfood, electron, slack, ...)

Step 2: the core loop

agent-browser open <url>
agent-browser snapshot -i              # accessibility tree, interactive elements only, @eN refs
agent-browser click @e12               # act on refs from the snapshot
agent-browser fill @e7 "text"
agent-browser screenshot /tmp/<name>.png
agent-browser close

• Re-run snapshot after every navigation or state change. Refs are only valid for the snapshot that produced them.
• Run one browser session at a time. Call store-progress between page batches so the heartbeat watchdog sees you.
• The browser is headless. Local servers inside the worker container are reachable at http://localhost:<port>.
• The image launches Chromium with --no-sandbox,--disable-dev-shm-usage through AGENT_BROWSER_ARGS. If you pass your own --args, include --no-sandbox again or Chrome will not start.

Recording

Pass record start an absolute path (/tmp/<topic>/raw.webm). A relative path resolves against the agent-browser daemon's cwd, not yours, and fails with ffmpeg write failed: Broken pipe while doctor stays green.

Default: record with --cursor, drive every supported pointer action (click, mouse move, drag) with --human, and ship the 1.5x video. Keep the 1x original only if a reviewer asks for it. Human pointer movement follows an eased curve; use --seed <n> on mouse move to make the take reproducible instead of using a bare move.

Open the page first, then record the interaction:

agent-browser record start ./in.mp4 --cursor
agent-browser snapshot -i
agent-browser mouse move 600 400 --human --seed 42 --duration 1400
agent-browser click @e3 --human          # use a ref from the current snapshot
# Perform the remaining QA steps; use --human for drag actions too.
agent-browser record stop
ffmpeg -y -i in.mp4 -filter:v "setpts=PTS/1.5,fps=30" -an \
  -c:v libx264 -preset veryfast -crf 23 -pix_fmt yuv420p out-1.5x.mp4

Speed-up is a post-process step: record start has no speed or human flag. Keep fps=30 to preserve a steady 30 fps output rather than uneven frame drops at an inferred rate. -an disables audio because agent-browser recordings carry no audio track. The worker ships only ffmpeg, not ffprobe; inspect duration with ffmpeg -i out-1.5x.mp4 2>&1 | grep Duration.

Add --contact-sheet to the start command for a timestamped PNG summary of visual changes. Recording needs ffmpeg on PATH; check agent-browser doctor before starting. Stop the recording before closing the browser so the video is saved. Keep the 1.5x file on the same agent-fs QA path as screenshots below. In a PR, upload it as a GitHub user attachment and put the URL on its own line, so GitHub plays it inline.

Step 3: share the screenshot through agent-fs

Screenshots on the worker disk disappear with the task. Upload them to agent-fs under the qa path convention: that copy is the durable original. A person opens it through the live app link ${AGENT_FS_LIVE_URL}/file/~/<org>/<drive>/<path>. --file is binary-safe. --content is text-only and mangles PNGs.

agent-fs write thoughts/<agent-id>/qa/<topic>-screenshots/<name>.png \
  --file /tmp/<name>.png -m "<what it shows>"
agent-fs stat thoughts/<agent-id>/qa/<topic>-screenshots/<name>.png --json        # confirm size > 0

• GitHub PR body or comment: upload the PNG, GIF or mp4 as a GitHub user attachment with the github-attach skill and embed ![caption](https://github.com/user-attachments/assets/<uuid>). Attachments never expire and mp4s play inline. A UI PR puts a before (main) and after pair in the ## Before / after table. Fallback for images only, when the GitHub token is an App token (ghu_/ghs_): agent-fs signed-url <path> --inline --json (expires in 7 days). Never a presigned URL for an mp4 or a doc: it downloads.
• Linear comment or Slack message: paste the live app link.
• In store-progress, list the upload in the attachments field with kind: "agent-fs" and the path, and paste the live app link for people who need to open the file.
• Make an agent-fs share-create <path> --json link only when the requester asks for one or the reader has no agent-fs login (customer, public post, email, named outsider). Never make one in case the file gets forwarded.
• If agent-fs auth whoami fails, report the local path, say the upload was skipped, and continue.

The agent-fs skill holds the full agent-fs recipe.

Gotchas

• wait <selector> waits for a VISIBLE match. When the first match is hidden it times out (about 25 s) even though the element exists. Wait on rendered text instead: wait --text "<rendered text>".
• Never run pkill -f <pattern> inside a Bash call: the pattern also matches the calling shell's command line and kills it. Record the PID when you start a background server (echo $!) and kill <pid>.