codename-11/hermes-relay/android

Android Phone Control

Drive a paired Android phone from Hermes using the android_* tool family. This skill is the playbook the LLM reads before touching those tools for the first time — architecture, pairing, the tool surface, bounded tool budgets, and per-app step-by-step procedures.

1. How it works

1. The Hermes-Relay Android app runs on the phone and exposes an AccessibilityService + MediaProjection dispatcher.
2. The phone connects outbound to the Hermes-Relay WSS relay (default :8767) as the bridge channel.
3. Each android_* tool sends an HTTP request to the relay on loopback, which forwards it as a bridge.command envelope over the WSS socket to the phone. The phone executes the action and replies with bridge.response.
4. Sensitive actions pass through Tier 5 safety rails on the phone (blocklist, destructive-verb confirmation, idle auto-disable). See section 9.
5. sideload builds unlock additional capabilities behind a build-flavor flag (voice-to-bridge intents, direct SMS, direct dial). See section 10.

2. Setup — how the phone gets paired

Do not call `android_setup`. That is a fallback for teaching the host about an existing session token — it does NOT pair a new phone. For first-time setup or re-pairing, use one of the canonical flows:

If the user has not paired yet, stop and hand off to /hermes-relay-pair. Do not attempt to drive tools against an unpaired phone — every call will return 503 bridge not connected.

To confirm the phone is reachable before starting a workflow, call android_ping once. A healthy response looks like {"status": "ok", "bridge": {...}}. Anything else means the phone is not connected — stop and tell the user.

3. Available tools

All of these live in plugin/tools/android_tool.py (except android_navigate in plugin/tools/android_navigate.py). Every tool returns a JSON string.

Screen reading

Interaction

App state

Orchestration

There is also an android_setup(bridge_session_token) function in the same module — that is a fallback for reconfiguring the host after a re-install when you already have a session token. It is not how you pair a phone. See section 2.

4. Critical: do not loop

Hard rule: cap any single user request at 5–7 tool calls total. When you hit that budget, STOP and report the current state to the user. Do not retry. Do not keep tapping. Do not "try one more thing."

Why this exists:

• Accessibility trees are noisy. The same visible "Send" button can have three different nodeIds across a single app depending on what's loaded above it. A naive retry loop burns minutes chasing stale node ids.
• Phones have Tier 5 safety rails (section 9). If a tap is blocked, retrying it will just produce the same 403. Report the block — don't hammer the gate.
• android_navigate is the correct escape hatch for "I don't know what to tap next." Its iteration cap is code-enforced. Your job as the outer agent is to bound the total number of invocations, not to write your own retry logic around it.

STOP-and-report pattern. When you cannot make progress:

1. Call android_read_screen once.
2. Summarize what you see (2–3 sentences).
3. Ask the user what they want you to do next. Do not guess.

Acceptable tool budgets for common workflows:

If you realize mid-workflow that you need more than 7 calls, stop, explain the situation to the user, and ask whether to continue.

5. Workflow pattern

The canonical shape of every bounded Android workflow:

open_app(package)
  → wait(text="<landmark>", timeout_ms=5000)
  → read_screen()
  → 1–3 targeted actions (tap_text / tap / type / swipe)
  → read_screen()        # verify the action landed
  → STOP and report

Key discipline:

• Verify before you chain. Never assume a tap "worked" — re-read the screen and confirm the next landmark is present before sending another action.
• Prefer `tap_text` over `tap(x, y)`. Coordinates go stale the instant the layout shifts (keyboard pops, ad loads, notification arrives). nodeId from a fresh read_screen is second-best; raw coordinates are last resort.
• Never `open_app` twice in a row. If the first didn't land, check android_current_app and report.
• Never chain more than 3 actions without a `read_screen`. UI state drifts faster than you expect.

6. When to use android_navigate vs direct tools

Use the direct tools (tap_text, type, swipe, etc.) when you know the exact step sequence — for example, the app playbooks in section 8. You are in control of the budget, you can inspect intermediate state, and every step is cheap.

Use `android_navigate(intent, max_iterations=N)` when you need vision in the loop: the user's goal is clear but the path is not, the layout is unfamiliar, or the target element is visible in pixels but not in the accessibility tree (icon-only buttons, canvas-rendered UI, custom views). android_navigate takes one screenshot per step, lets a vision model pick the next action, and dispatches to the same direct tools under the hood. The iteration cap is enforced in code (default 5, max 20).

Rules of thumb:

• Known app, known steps → direct tools. (Uber, WhatsApp, Spotify, Maps — see section 8.)
• Unknown app or UI → `android_navigate`. Let vision decide.
• Mixed → direct tools to get into the right screen, then `android_navigate` for the last-mile decision. Don't burn vision budget on open_app → wait that you can script.
• Never wrap `android_navigate` in your own retry loop. If it fails, read the trace, summarize for the user, and stop.

7. Common package names

If the user asks about an app not on this list, call android_get_apps and match on the human-readable label.

8. App playbooks

Each playbook is a minimal bounded sequence. Follow them; do not improvise extra steps.

Uber — book a ride to a known destination

1. android_open_app("com.ubercab")
2. android_wait(text="Where to", timeout_ms=8000) — Uber's home search field.
3. android_tap_text("Where to") — opens the destination picker.
4. android_type("<destination>") — type the address into the focused field.
5. android_wait(text="<destination-first-word>", timeout_ms=4000) — wait for autocomplete.
6. android_tap_text("<first matching result>") — select the result.
7. android_read_screen() — verify the ride-type picker appeared. STOP and report the quoted fare + ETA. Do not auto-confirm the ride — requesting a ride is a destructive action that spends money. Let the user confirm manually, or let the Tier 5 destructive-verb gate (section 9) prompt them.

Pitfalls:

• Uber sometimes wraps the destination field in a non-clickable container. If tap_text("Where to") does nothing, re-read the screen and look for a clickable ancestor node.
• If a login or onboarding splash appears, stop and report — do not try to log in for the user.
• The "Confirm" / "Request" button is on the destructive-verb list by default (see section 9). Expect a 403 if you try to tap it without user confirmation.

WhatsApp — send a message to a contact

1. android_open_app("com.whatsapp")
2. android_wait(text="Chats", timeout_ms=5000) — landing tab.
3. android_tap_text("Search") or tap the magnifying glass icon.
4. android_type("<contact name>") — narrows the chat list.
5. android_tap_text("<contact name>") — opens the chat.
6. android_tap_text("Message") — focus the input field. (Label varies by locale; fall back to read_screen and find the EditText node.)
7. android_type("<message body>")
8. STOP and report. Do not auto-tap Send — it is on the destructive-verb list. Either let the user tap Send, or invoke the Tier 5 confirmation modal by calling android_tap_text("Send") and letting the phone prompt.

Pitfalls: