mingye-lu/agenticcrawler/acrawl-mcp

acrawl MCP

acrawl exposes 40 MCP tools: 32 browser/DevTools tools, 7 script tools, and 1 autonomous agent (run_goal). All 40 share one persistent browser session across calls — except run_goal, which builds its own isolated agent and browser.

What makes acrawl different from other browser MCPs is the observability half: you can read network traffic, console logs, WebSocket frames, cookies, storage, coverage, and axe-core audits from a live page. Reach for those instead of guessing why a page misbehaves.

Tool names below are bare (navigate). Your client may namespace them (acrawl_navigate, mcp__acrawl__navigate) — use whatever prefix your tool list shows.

1. Pick an execution mode first

Three ways to get work done. Choosing wrong is the most expensive mistake available, because it either burns tokens on a loop that should have been a script, or spends a script debugging cycle on a one-off click.

The practical pattern: explore manually until the shape is known, batch the short predictable sequences, then script the repetition. Do one page by hand, confirm your selectors, then turn the loop into a script. A 50-page crawl becomes one tool call instead of 150.

Reach for run_goal when the target is genuinely unknown and adaptive ("find the pricing page and extract every tier"). It needs ~/.acrawl/credentials.json configured; the other 38 tools do not. If credentials are missing, run_goal fails and the correct move is manual tools or a script, not repeated retries. See references/setup.md.

2. The core loop: observe, act, verify

navigate → page_map → act (click/fill/select) → read the returned page_state → repeat

Every interaction tool (click, click_at, fill_form, hover, press_key, select_option, scroll, go_back, switch_tab, set_device, refresh) returns a page_state for free. Read it instead of calling `page_map` again — that's a wasted round-trip and a wasted page render.

page_state arrives in one of three shapes:

• Full — first interaction on a URL, or when changes are too broad to diff.
• Diff — changed: true plus a changes block using + added / - removed / ~ state-changed. This is the common case and it is small; trust it.
• No-op — changed: false. Your action did nothing. Do not retry the identical call; the selector matched nothing meaningful, the element was disabled, or a guard blocked it. Re-map the region and pick a different target.

A changed: false after a form submit deserves suspicion: acrawl reports a CaptchaDetected error for likely-silent reCAPTCHA v3 rejections, but a plain no-op can also mean client-side validation failed. Check list_page_logs for the reason rather than resubmitting.

Batch predictable steps into one turn

Emit several tool calls in a single turn whenever you can predict the next steps. Every turn you save is one fewer LLM round-trip, and this is the cheapest speed-up available — it needs no script and no setup. The default of "one call, read the result, decide" is right for exploring an unfamiliar page and wasteful everywhere else.

Batch these:

• Independent reads of the same page — several read_content selectors, or read_content + list_resources.
• Read-only follow-ups to one state-changing action — scroll → read_content, or press_key → wait → read_content.
• Observability sweeps — list_network_activity + list_page_logs + list_websocket_activity after one action.

Don't batch these:

• Anything that consumes an earlier call's output in the same turn. @eN refs go stale after a navigation or substantial DOM change, and refs from page_map don't exist until it returns. page_map → click @e7 cannot be one batch. Neither can a listing followed by an inspector on one of its refs.
• A submit behind a form update. Never queue click on a submit button after fill_form or select_option: if the update fails or only partly applies, the submit sends empty or stale values. Run the update, read its page_state, then submit in the next turn.
• Actions after a step that may change the page unpredictably (a click that might open a modal or show validation errors).

Rules of thumb:

• Order for safe failure. Reads first, state-changing calls (click, fill_form, navigate) last. Never queue an irreversible action behind a step that might fail.
• Failures may not stop the queue. Whether later calls still run after an earlier one errors is up to your MCP client; acrawl executes each call it receives. After a batch, check every result and re-plan from the actual page state if any failed.
• Check your client's ordering. acrawl shares one browser session across calls and does no ordering of its own. If your client dispatches a turn's calls concurrently instead of in order, restrict batches to independent reads.
• Keep batches to about 2–5 calls. Each result lands in context, so avoid queueing several large read_content calls.
• 3+ repeats of the same operation is a script, not a batch. See section 1.

3. Token discipline

This is where acrawl sessions go wrong. A full navigate on a content-heavy page plus a deep page_map can dominate your context in two calls. Defaults are already tuned for economy — the failure mode is overriding them upward "just in case."

`navigate` gives you two independent dials. They compose, and both matter:

• content_depth: main (default) · full · slim (first 2000 chars) · none
• page_map_depth: slim (default, omits CSS selectors) · full · none

Set content_depth: "none" whenever you only need structure to click through — you get the map without the prose. Set page_map_depth: "none" when you only want text. Setting both to none makes navigate a cheap "just load it" call, which is exactly what you want mid-script.

`format` controls prose conversion. fit_markdown (default) prunes boilerplate for roughly 30–60% savings by removing elements whose class/id contains nav, footer, header, sidebar, ads, comment, promo, advert, social, share, then score-pruning the rest. Switch to markdown when the content you actually want lives in a sidebar or nav — author bios, metadata panels, related links. That's the one case where the default actively destroys your target.

Other habits that pay:

• Prefer read_content with a heading or selector over re-navigating. It paginates via offset/max_chars (default 10000), so pull large sections in slices.
• Scope page_map with scope ([ref=eN], or a semantic token like dialog/main/sidebar) and keep depth shallow. Default depth 5 is usually plenty; max is 10.
• wait with seconds and silent: true skips the page_state diff — use it for pure pauses.
• `screenshot` is a last resort. It cannot be searched, diffed, or parsed, and it is expensive. Text tools answer almost every question you would take a screenshot to answer. Use it for genuine visual verification only.
• list_resources returns all links/images/forms with no cap — it exists precisely because page_map truncates links at 50. Use it when you need the complete set, not as a general explorer.

4. The @ref system

acrawl hands out short opaque handles instead of making you carry selectors or URLs. Four families, each produced by a lister and consumed by an inspector:

The list→inspect split is deliberate: listing stays cheap and scannable, and you pay for detail only on the one item you care about. Don't ask for detail on everything.

`@eN` refs go stale. They are tied to a page render. After a navigation or a substantial DOM change, re-read them from the fresh page_state or page_map. A stale @eN is a common cause of a mystifying changed: false.

`@rN` / `@logN` / `@wsN` refs are rebound on every list call. They are positional within the most recent listing, not stable global IDs — despite the list_network_activity description saying "stable @rN refs." Verified: @r1 was a font file under sort_by: ["largest"], then became a failed script under filter: "failed", and inspect_request(@r1) returned the latter. So inspect immediately after the listing that produced the ref, and never carry a ref across two listings with different filter/sort_by/since.

When CSS paths are fragile, target by label instead. click accepts text (optionally narrowed by role and region) and fill_form resolves field identifiers by visible label text page-wide — including inside modals and div-based UIs with no <form> element. This is usually more durable than a generated class path in an SPA admin panel.

select_option has a discovery mode worth knowing: call it with only selector and no value/label/index, and it opens the dropdown and enumerates the available options without selecting anything. Use it before guessing.

5. Temporal filtering for observability

The DevTools tools buffer continuously from browser launch and share a `seq` counter that increments on every action. This is what makes "what did this click do to the network?" answerable.

list_network_activity, list_page_logs, and list_websocket_activity all take:

• since: "last" (default — since your last action) · "all" (whole session) · a numeric seq
• until: "now" (default) · a numeric seq

since: "last" is the default because attributing effects to the action you just took is the common question. Use since: "all" to retrieve requests from before you started looking — capture begins at browser launch, so early page loads are still there.

Numeric `since` is inclusive, and an action's observations carry the same seq that action's own response returns — the window is [since, until). So to isolate one interaction, perform it first, then query with the seq it just returned. Anchoring on the previous action's seq sweeps that action's observations in too.

Verified: two navigations returned seq: 0 then seq: 1; since: 0 listed 64 requests including https://example.com/, while since: 1 listed 63 and excluded it.

list_network_activity sorts by adjective — sort_by: ["slowest", "largest"] (first is primary, rest are tiebreakers) — and filters by filter (xhr/media/failed/pending/aborted), pattern, method, and size bounds. unique_urls: true collapses repeats, keeping the largest response as representative. That combination answers "what is making this page slow" in a single call.

Interception requires a reload to take effect. intercept_network rules are additive and apply to future requests, so the sequence is: add rule(s) → refresh → observe. Skipping the refresh is the usual reason a block appears to do nothing. Clean up with action: "clear_all".

6. Verified gotchas