name: joshu-browser-handoff description: Live browser HITL — hand the shared browser tab to the owner on mobile for login, payment, 2FA, or owner-only steps. metadata: version: "1.0.1" hermes: category: browser version: "1.10.0"

Joshu browser handoff (live HITL)

Use when the owner must take over the shared browser tab on their phone — payment, login, 2FA, irreversible confirms, or any step that is sensitive, policy-bound, or explicitly owner-only.

This is sustained live HITL: the owner gets a signed link, the live tab (Browser Use Cloud iframe on fleet boxes, CDP screencast on local Chromium), your brief, the form overlay, and time to finish. It is not a single-click SMS approval (see action guard below).

Handoff tools: browser_handoff_request, browser_handoff_status, browser_handoff_complete (plugin joshu-browser-handoff).

Owner SMS: any inbound owner text auto-completes a pending handoff for that SMS session before your turn (browser already unlocked). jChat/voice: call `browser_handoff_complete` when the owner says they are finished — no need to tap I'm done on the link.

Browser backend (read this first)

Joshu boxes use one shared browser session. The backend is chosen at stack start (Safety Settings or JOSHU_CLOUD_BROWSER in instance.env). Never mix backends in one task.

Cloud fleet rule: stage pages with `browser_navigate` only, then `browser_handoff_request`. Handoff reads the same cloud CDP session Hermes used. If you get no_active_browser_tab, `browser_navigate` again to the handoff URL on the same Hermes browser tools — do not fall back to browser_task or to localhost browser REST.

Local rule: call `browser_task` for autonomous browsing. Stop before payment, CAPTCHA, or secrets, then `browser_handoff_request`. While handoff is pending, browser_task is refused.

Browser down = system problem. Joshu wakes the shared browser automatically on every browser_* call. If a browser tool still fails with `browser_unavailable` or a CDP / WebSocket error (e.g. 502 Bad Gateway): retry that tool once, then `kanban_block("system: browser unavailable")` and stop. Never investigate the box — no ps / ss / port scans, no reading /opt/joshu/dist or Hermes source, no changing Hermes config (hermes config, config.yaml), no curling local Joshu or browser supervisor APIs to drive the page. Joshu owns that recovery.

When to use

Rule of thumb: if the next meaningful browser actions belong to the owner (credentials, money, or binding consent), stage the page and hand off.

Never ask the owner to send a secret

Never ask the owner to send a password, passcode, PIN, recovery code, or one-time code by SMS, chat, email, or voice, and never offer it as an option ("reply with the code and I'll type it in"). A secret in a text stays in the SMS log, the Hermes session, and the traces.

When a page asks for a secret (a sign-in, a 2-factor code, the passcode of encrypted direct messages, a PIN), do this at once, with no question first:

1. Stage the page that asks for the secret.
2. Call `browser_handoff_request` with a brief that says what to type and where, for example "X asks for your 4-digit passcode for direct messages. Type it in the boxes, then tap I'm done."
3. Send the handoff link.

If the owner sends a secret anyway, do not repeat it in any message, and use it only for the step that the owner asked for.

Account-specific web data (Amazon orders, bank, portals, RapidAPI)

Use the browser + handoff — not Composio/Gmail search alone — when the answer lives behind the owner's login:

Gmail may help after the fact (confirmation emails), but it is not a substitute for an authenticated session.

SMS / owner texted you

When the channel is SMS, still `browser_handoff_request` and include the full handoff URL in your reply. Joshu splits long texts across multiple SMS — do not omit the link to stay under 500 characters.

After handoff on SMS: browser_snapshot, then put the answer in your assistant reply text (order details, confirmation, etc.). Joshu sends that as SMS automatically.

Never call nylas_send_message or email the owner to deliver SMS-originated handoff results — that triggers action-guard approval and skips the SMS reply path.

Workflow

1. Do prep work autonomously — search, compare, fill non-sensitive fields, verify constraints in `browser_snapshot` (dates, price, terms, correct account, etc.).
2. `browser_navigate` (cloud) or `browser_task` then navigate (local) to the page where the owner should start.
3. `browser_snapshot` — confirm the staged state matches what you will describe in the brief.
4. `browser_handoff_request(instructions=…)` — short owner brief: what site, what to check, what to do, any caps or policies.
5. Include the returned `url` in your outbound message (email, jChat, or SMS). On SMS, always send the link — splitting is automatic.
6. `kanban_block(reason="awaiting owner browser handoff", kind="needs_input")` (or equivalent wait state). kind="needs_input" makes the task an owner question, so the owner can get a proactive nudge.
7. Poll `browser_handoff_status(handoff_id=…)` until status is completed, or call `browser_handoff_complete` as soon as the owner says they are done (SMS/jChat/voice). Read the `message` field of the result: it says in plain words whether the staged page is still there. Joshu keeps the shared browser with the staged page until the link expires while the owner has not opened the link; after the owner opens it and leaves it idle for 10 minutes, or after the link expires, Joshu stops the shared browser and the staged page is gone.
8. `browser_snapshot` — verify the outcome (confirmation page, logged-in state, success message).
9. Deliver results on the same channel — SMS → assistant reply text only; email/jChat → normal outbound for that channel.
10. `kanban_complete` or continue the task.

During pending handoff, do not navigate away — the server pins pageUrl and blocks agent writes.

Handoff vs action guard

Cold browser (local Chromium only)

On local Chromium boxes, the browser may have no tab after a restart. Before the first heavy browser_navigate or browser_task:

curl -fsS -X POST http://127.0.0.1:8788/joshu/api/camofox/warm

Cloud fleet boxes skip this — the first browser_* call wakes (or recreates) the Browser Use session itself. Logins saved to the browser profile usually survive a recreate; open tabs, carts, and half-filled forms do not — re-navigate.

Owner experience

Owner opens the handoff link on their phone → signs in with the box username and password (a desktop session is not enough) → compact Joshu header + embedded live browser → types in native fields (Fill enables after an edit) or More Options for scan/paste → I'm done (bottom right).

On a local box, desktop jWeb and the phone can show the CDP screencast at the same time. The shared page takes the shape of the viewer that connected or resized last.

Errors

Never patch this skill with browser REST workarounds after a handoff failure — that indicates an infrastructure mismatch, not missing procedure.

Related

• Browser backends: `docs/hitl-camofox-notes.md`
• Action guard (single-click HITL): `docs/agent-safety.md`
• Travel checkout after owner picks: `realtime-goal` booking phase