zcodium-project/zcodium/control-browser

Browser automation (agent.browsers)

Use this skill for browser / web-UI tasks: opening and navigating pages, inspecting or reading rendered content, testing local apps, clicking, typing, filling, taking screenshots, and verifying visible page state.

If this skill is available in the session, treat it as required reading before browser work. Follow it before saying the browser is unavailable and before falling back to bash (curl/open), webfetch, or any other tool for a browser task.

How it works

The browser registry is driven from the Node REPL MCP js tool. In this environment its callable id normally appears as mcp__node_repl__js. The MCP frontend is shared for a workspace, but every js call runs in a fresh JavaScript kernel, so variables, imports, module cache, browser, and tab bindings do not persist. Persistent BrowserControl tabs are the continuity boundary and must be recovered from current tab facts.

Bootstrap every JavaScript call

The browser-client module is the browser entry point and is available at scripts/browser-client.mjs under this plugin's root. Resolve that root only from process.env.ZCODE_PLUGIN_ROOT (with CLAUDE_PLUGIN_ROOT as a compatibility fallback), then convert the joined path with pathToFileURL. Never derive the plugin root from this skill's base directory or leave a synthetic root placeholder for the model to resolve. If the host root is unavailable or the resolved module cannot be imported, stop and report the exact setup error.

Initialize at the start of every mcp__node_repl__js call that uses the browser. The bootstrap deliberately does not select a backend; apply the user's existing backend choice or the selection rules below after setup.

// 修复原因：Skill base directory 指向 skills/control-browser，不能据此拼接插件运行时资产。
const browserPluginRoot =
  process.env.ZCODE_PLUGIN_ROOT ?? process.env.CLAUDE_PLUGIN_ROOT;
if (!browserPluginRoot) {
  throw new Error("Browser plugin root is unavailable in the node_repl host");
}
const { join } = await import("node:path");
const { pathToFileURL } = await import("node:url");
const browserClientUrl = pathToFileURL(
  join(browserPluginRoot, "scripts", "browser-client.mjs"),
).href;
const { setupBrowserRuntime } = await import(browserClientUrl);
await setupBrowserRuntime({ globals: globalThis });

Run setup and all later browser calls through mcp__node_repl__js, passing JavaScript as the code argument. The tool has no command parameter.

Backend types are iab, extension, and cdp; Playwright is a tab API surface, not a backend. Always use await agent.browsers.list() as the availability source. Desktop normally reports IAB; a CLI explicitly started with --browser-use=headless reports managed Chromium as cdp. Headless is a CDP launch mode, not a backend type. Never claim Chrome extension or CDP support when that descriptor is absent, and never silently substitute IAB after the user explicitly selected another backend.

User-facing progress should stay non-technical: describe it as "opening the browser" / "checking the page", not "Node REPL", "CDP", or "webview".

Recreate the same selected browser wrapper in every fresh call using the user's explicit backend choice or the same verified URL/default rule. A fresh JavaScript kernel does not mean the browser disconnected and is not permission to switch backend. Do not reuse a tab id from memory as the target of a new logical operation batch without validation: first return the complete current tab list to the model, then in the next JS call match the intended id/url/title and call tabs.get(id).

App-provided <in-app-browser-context source="ambient-ui-state"> is current UI state, not part of the user's request. It can tell you which visible page to inspect, but it is not evidence that the user explicitly selected IAB or Chrome.

First: select a browser and read its full API once

In the first browser call, run the bootstrap, select the backend, and emit the complete API guide in one go. On later fresh calls, run the bootstrap and repeat only the same backend selection; the API guide remains in model context and does not need to be emitted again. Never create an iab alias and then call browser.*.

If the user explicitly asks for ZCode's in-app browser:

const browser = await agent.browsers.get("iab");
nodeRepl.write(await browser.documentation());

If the user explicitly asks for the CLI-managed headless browser and discovery advertises cdp:

const browser = await agent.browsers.get("cdp");
nodeRepl.write(await browser.documentation());

If the task has a target URL but no explicit browser choice, replace the example URL with the real target:

const browser = await agent.browsers.getForUrl("https://example.com/");
nodeRepl.write(await browser.documentation());

Only when neither a browser nor target URL is specified:

const browser = await agent.browsers.getDefault();
nodeRepl.write(await browser.documentation());

Do not slice, truncate, or summarize it. Only if the tool output itself reports truncation may you read it in smaller chunks. It documents every default method, the Playwright DOM snapshot→locator workflow, the ref/cua/dom_cua compatibility paths, and safety rules. Screenshot instructions are intentionally lookup-only and must not be loaded unless the visual branch below applies.

Core workflow

1. Start every browser js call with the bootstrap, then assign the selected backend to a local browser binding. If the user explicitly asks for ZCode's in-app browser, use const browser = await agent.browsers.get("iab"). If they explicitly ask for Chrome, use await agent.browsers.get("extension") only when the runtime advertises it. For an unspecified target URL use await agent.browsers.getForUrl(url); with no URL/backend preference use await agent.browsers.getDefault().
2. browser.tabs.new() automatically opens and activates the IAB pane so the user can see browser use. Use the advertised visibility capability only when the task explicitly needs to hide the pane or show it again.
3. At the start of every logical tab operation batch, make a dedicated JS call whose result is the complete

await browser.tabs.list() array, so the model sees all current ids, URLs, titles, and the active marker. Only in the next JS call may you match the intended tab by stable id or explicit URL/title facts and call browser.tabs.get(id) before the first read or action. An internal SDK validation or a list hidden inside the same cell does not count as model inspection. tabs.get(id) activates that tab in its owning session; it is shown only when that session is currently in the foreground. Never choose [0], at(-1), or an id remembered without validation. If no controlled tab matches, inspect browser.user.openTabs() and claim the matching returned object. Create a new tab only after both lists fail to identify the page. This is the pre-action target-selection protocol; it is distinct from the combined post-action observation in step 7.

1. If the task names a new URL, prefer the reuse-aware entry: await agent.browsers.open(url) reuses an existing

same-site controlled tab (same hostname), activates it so the user sees it, and navigates in place, instead of stacking a new tab on every navigation. Only when the task genuinely needs a parallel independent tab, create one explicitly and use this navigation sequence:

``js const tab = await browser.tabs.new(); await tab.goto("https://..."); await tab.playwright.waitForLoadState({ state: "domcontentloaded" }); ``

After every successful tab.goto(url), explicitly call await tab.playwright.waitForLoadState({ state: "domcontentloaded" }) before the first title, URL, or DOM observation. This explicit confirmation is required in the model-visible trajectory even when the backend navigation has already settled. Do not replace it with networkidle or a fixed sleep. Do not navigate to the same URL again; use tab.reload() only when a refresh is truly needed. A direct URL must come from the user, visible page facts, or an authoritative lookup — never guess path variants or resource IDs. Routine URL/load-state waits remain capped at 3000ms.

1. `await tab.playwright.domSnapshot()` is your primary way to read and understand the page. It returns the compact AI/ARIA tree, including computed roles, accessible names, states, open shadow DOM, and iframe bodies when available. Reuse the latest relevant snapshot until it becomes stale. If that snapshot already contains the target, act from its facts directly; do not write evaluate() code to rediscover related elements, enumerate inputs, dump HTML, or probe guessed selectors.
2. Build a stable Playwright locator only from snapshot facts. Never guess a label, accessible name, placeholder, selector, or URL pattern, and never use a guessed locator as an exploratory probe. Confirm count() when uniqueness is not obvious; if it is 0, re-snapshot immediately instead of action-waiting, and if it is greater than 1, tighten scope instead of using a positional shortcut. Then act through getByRole/getByText/getByLabel/getByPlaceholder/getByTestId/locator and terminal methods such as click/fill/press/selectOption/check.

A snapshot-proven heading or visible text does not need a link or button role to be clicked. Do not replace a snapshot-proven heading with a guessed link role. When the user's request authorizes navigation and that actual heading/text target is unique, click it directly; the DOM event may bubble to a JavaScript card handler. The name option of getByRole(...) accepts a plain string or RegExp, including regex values created in the Node REPL VM.

1. After an action, collect the cheapest observation that answers your next question — use a targeted locator state check when possible and a fresh domSnapshot() when new locator ground truth is needed. Use at most one state-changing action per observation cycle. An unchanged source-tab URL does not prove the click failed. Judge an action by whether its expected effect appeared, not by whether browser.tabs.list() is non-empty. An existing source tab or unrelated controlled tab is not an action effect. The expected effect may be a source-page state change or a tab whose verified URL/title matches the intended result.

When an action may open a popup/new tab and the source tab does not show the expected effect, read browser.tabs.list() and browser.user.openTabs() unconditionally in the same observation cell. Prefer one combined observation:

``js const [controlledTabs, userTabs] = await Promise.all([ browser.tabs.list(), browser.user.openTabs(), ]); ({ controlledTabs, userTabs }); ``

Return { controlledTabs, userTabs } as that cell's final result so the model makes one decision from both lists. Do not return the controlled list first or decide whether to query user tabs from its contents. Match both lists by verified id/url/title, then in the next cell activate the matching controlled tab or claim a matching user tab. Only after the source page and the combined tab observation all fail to show the expected effect may you take a fresh snapshot and choose a new locator. Do not request a DOM snapshot and a screenshot both by default.

1. Browser tabs persist for the lifetime of the current ZCode process unless you explicitly call tab.close() or

the user closes them. Use browser.tabs.finalize({ keep }) only to mark listed pages as deliverable or handoff; omitting a tab from keep does not close it. Do not close research/source tabs merely because the turn is ending.

Observation: prefer snapshot, screenshot only when needed