coder/xum/bug-bash

Bug bash in the sandbox

Each charter is one e2e explore run: an AI explorer drives Chromium against its own seeded xum server and reports findings. Every model-driven run happens inside the bug-bash sandbox (#5714), a Docker container per job. Findings are claims. Report a bug only after a repro test fails for the reported reason.

The generic method (charters, triage, verify, report) is upstream's guide: node_modules/.bin/e2e guide bug-bash (it needs Node.js 22.22.3+ or 24.8+). This skill adds the Xum harness. Where the two differ, follow this skill.

How the sandbox works

• The host pause (tests/bugbash/hostPause.ts) refuses every model-driven e2e process on the host

with exit 2: e2e explore, and every command with e2e.mcpapps.config.ts, tests/stories/e2e.config.ts or tests/stories/e2e.electron.config.ts. Outside the sandbox the agents hold no model, so an agent.* step fails with MODEL_UNAVAILABLE. No override exists. e2e run and e2e list of exact-step repros still run on the host.

• make bug-bash (tests/bugbash/run.ts), make mcp-apps-e2e and make test-stories are the

launchers. They run tests/bugbash/sandbox/launch.ts, one container per job.

• The image ghcr.io/coder/xum-bugbash-sandbox (about 1.4 GB, linux/amd64) holds only tools:

Node 22, Bun, git, the Chromium of @e2e-dev/web's playwright-core, and Xvfb plus libgtk-3 for Xum Electron (the Electron binary comes from the mounted node_modules). The runner pulls it only by the digest in tests/bugbash/sandbox/image.json. It builds it only in the case below.

• The inputs key (make bugbash-sandbox-key) is a hash of the Dockerfile, build.sh and the

playwright-core version in bun.lock. The runner refuses an image whose key differs from the checkout. One exception: with BUGBASH_SANDBOX_BUILD_STALE=1 (exactly 1), which only CI's Test / Bug-bash sandbox job sets, the self-check and the exact-step repros build the image locally with build.sh (local tag, no push, no registry login), check its inputs label, and print a warning (a ::warning:: annotation in GitHub Actions). Explore charters, the MCP Apps suite, the story tests, the agent repros and the preflight of make bug-bash, make mcp-apps-e2e and make test-stories always refuse a stale key.

• An image change lands in three steps: the PR that changes the Dockerfile, build.sh or

playwright-core passes CI on a local build. After it merges, a maintainer runs the manual workflow "Bug-bash sandbox image" on main. Then a reviewed PR updates image.json to pin the new digest. An e2e update that keeps playwright-core keeps the key, so the image stays valid.

• The container gets --network none, --read-only, --cap-drop ALL, no-new-privileges, a

non-root user, 4 GB memory and tmpfs scratch folders. It reads copies of the git-listed files, plus dist/ and node_modules/ read-only. An env allowlist passes no credential.

• One provider proxy per job (sandbox/proxy.ts) runs in the launcher on the host and holds the

API key. The container reaches it only through a read-only unix socket. The proxy allows only POST /anthropic/v1/messages to the job's models, clamps max_tokens, and reserves each call's worst-case cost from the run's ledger before it sends the call. It writes one record per call to <output>.proxy.jsonl.

• The container writes nothing on the host. At the end, sandbox/entry.ts sends the output folder

on stdout as framed files (sandbox/exportStream.ts). The host writes only new regular files and stops at the first bad frame.

• The container's stdin is the lifeline: when the launcher dies, the container stops.

1. Prepare

1. Host: Linux on amd64, not root, with a local rootful Docker socket. Docker Desktop, rootless

Docker and a remote endpoint refuse (exit 2). The host needs git, make and the Bun version of packageManager in package.json (bun --version): make builds with Bun and runs the launchers with it. It needs no Node.js or Chromium for the sandbox targets. On a fresh checkout, run bun install.

1. Env: ANTHROPIC_API_KEY, ANTHROPIC_BASE_URL (with or without /v1) and

BUGBASH_BUDGET_USD. The runner reads the key only from ANTHROPIC_API_KEY: copy a key from another variable (for example ANTHROPIC_AUTH_TOKEN) there.

1. BUGBASH_BUDGET_USD caps the whole run (every job and the app-AI probe) at list price. It is

not a billing cap: the key's own limit is the backstop.

1. Dry run at $0 with no key: make bug-bash-sandbox-check (isolation, proxy and export

refusals against a fake upstream, about 2 minutes) and make test-bugbash-repros-sandbox (the repros through the launcher on the mock app AI, about 4 minutes). CI's Test / Bug-bash sandbox job runs both, in the merge queue and on PRs that change tests/e2e, tests/bugbash, playwright.config.ts, pr.yml, Makefile, package.json or bun.lock.

2. Plan charters

Edit tests/bugbash/charters.txt, one slug|target|agent|charter per line. One area and one persona per charter. For a branch, base the charters on git diff --stat origin/main...HEAD, or pass a branch-specific file with --charters <file>.

• Targets (tests/bugbash/e2e.config.ts): web 1440x900, phone 390x844.
• Personas (--agent): newcomer, keyboard, fuzzer, state, skeptic, or e2e's own default.
• Name the workspace ("Bug bash playground") and the starting view in the goal.
• Model-driven jobs turn terminals off, so no charter can use a Terminal tab. Settings can still

add stdio MCP servers and custom editor commands: keep explorers out of those.

3. Run

make bug-bash BUGBASH_BUDGET_USD=9                                  # 6 charters x Sonnet 5.5
make bug-bash BUGBASH_BUDGET_USD=3 BUGBASH_ARGS="--only composer"   # one charter

1. The run checks the env, Docker and the image first (about a second once the image is cached),

then builds, then launches the jobs. The first run pulls the image, which takes minutes.

1. Before any job it prints its estimate and a recommended budget, and refuses a lower budget

with the number to set. The default needs $9 and takes about 5 minutes after the build.

1. The app AI (tests/bugbash/aiMode.ts): BUGBASH_AI=auto (default) probes

BUGBASH_APP_MODEL (default anthropic:claude-haiku-5-5) once through a proxy. If the probe answers, the app uses that model through each job's proxy. If the upstream is busy (429 or 5xx), the app uses the mock. A rejected key (401 or 403) or an unreachable base URL refuses before any job. real never falls back, and mock never calls a provider. Charters that name a [mock:...] prompt always use the mock.

Options:

• BUGBASH_ARGS: --only <slug,...>, --charters <file>, --parallel N (default 8, all models

together), --max-steps N (1-12, default 6), --config tests/bugbash/e2e.mcpapps.config.ts (the only other config that runs).

• BUGBASH_MODELS: comma-separated anthropic:<model>, default Sonnet 5.5. Only models with a

price in tests/bugbash/sandbox/proxyPolicy.ts run (claude-sonnet-5-5, claude-opus-5-5, claude-haiku-5-5). Adding Opus finds more distinct bugs and needs about $29.

• BUGBASH_EFFORT: low, medium (default), high, xhigh or max. A higher effort raises the

estimate as a marked guess.

• make bug-bash refuses BUGBASH_MODEL, which only make mcp-apps-e2e and make test-stories read.
• Every model-driven job refuses BUGBASH_SCENARIO: its bash-ai-proxy scenario needs project

automation, which those jobs turn off.

MCP Apps: make mcp-apps-e2e BUGBASH_BUDGET_USD=5 runs tests/bugbash/mcpapps/mcp-apps.e2e.ts (7 tests on web and phone) with BUGBASH_MODEL (default Sonnet 5.5). It pins the mock app AI, because its seed writes the MCP chat directly. Measured runs spent $0.93 to $1.71 and took 8 to 10 minutes. MCP_APPS_E2E_ARGS passes e2e arguments. To explore it, use BUGBASH_ARGS="--config tests/bugbash/e2e.mcpapps.config.ts --charters tests/bugbash/mcpapps/charters.txt". Suite tests drive flows with agent.act in on-screen words and pin each outcome with an exact expect. Use agent.assert(..., { vision: "only" }) for "is it on screen": the accessibility tree also lists off-screen nodes. Prove a new suite test fails on the old dist/ before the fix lands.