jianzhichun/emerge/remote-runner-dev

Remote Runner Dev

Overview

Emerge's remote_runner.py is a lightweight HTTP execution server that runs on any machine where the actual tools live — a Windows VM with CAD software, a Linux HPC node with simulation tools, a GPU workstation, a mobile device emulator host. The dev machine stays clean; all environment-specific execution happens on the runner.

Core principle: deploy via `icc_exec` HTTP — not SSH exec, not SCP. The runner is its own deployment channel.

Architecture

Dev machine                       Remote machine (any OS)
────────────────                  ──────────────────────────────────
emerge project dir    deploy ─►   <plugin_root>/scripts/
  scripts/           icc_exec      runner_watchdog.py  (daemon)
  connectors/          HTTP ─►       └─ remote_runner.py (port <N>)
  scripts/repl_admin.py                   └─ executes icc_exec calls
                                          └─ has access to local tools,
                                             GUI, COM objects, hardware

The runner executes code in its own process environment — not in the SSH session that deployed it. This distinction matters when the tool requires:

• A specific user session (Windows COM, GUI automation)
• Local hardware (GPU, USB device, simulator)
• Environment variables set at desktop login (conda env, license server, display)

Key Source Files

Runner setup (first time)

Operators install from the daemon using one copy-paste command (no SSH from the dev machine).

python3 scripts/repl_admin.py runner-install-url --target-profile <profile-key> --pretty

Or open Cockpit → Monitors → Add Runner and copy the curl … | bash / irm … | iex lines. The script downloads the runner bundle from the daemon, writes ~/.emerge/runner-config.json, installs optional pip deps, and registers autostart (systemd / launchd / Windows Run).

After the operator has run the installer, persist the route on the dev machine if needed:

python3 scripts/repl_admin.py runner-config-set --runner-key <profile-key> --runner-url http://<host>:8787

Use runner-deploy for subsequent code updates (HTTP push, hot-reload).

Development Workflow

Edit → Deploy → Verify

# 1. Edit code locally
vim scripts/remote_runner.py
vim connectors/<vertical>/pipelines/read/state.py
 
# 2. Deploy (runner hot-reloads)
python3 scripts/repl_admin.py runner-deploy --target-profile <profile>
 
# 3. Verify
python3 scripts/repl_admin.py runner-status --pretty

How runner-deploy works

1. Reads scripts/*.py from local project
2. Base64-encodes each file, POSTs to <runner_url>/run as icc_exec calls
3. Runner writes files to <plugin_root>/scripts/ in-place
4. Touches .watchdog-restart → watchdog restarts runner with new code (~5s)

Why icc_exec, not SSH/SCP

Watchdog

runner_watchdog.py wraps the runner process:

• Crash recovery: if runner exits unexpectedly, restarts after RESTART_DELAY_S (default 3s)
• Signal restart: polls for .watchdog-restart file every POLL_INTERVAL_S (default 2s)
• Log: writes start/restart events with PID to .watchdog.log

Trigger hot-reload manually (without full deploy):

# via icc_exec to the runner
import pathlib
pathlib.Path('<plugin_root>/.watchdog-restart').touch()

Platform-Specific Setup Notes

Windows — interactive session required for GUI/COM

COM objects (DesktopDraftingApp, DesktopDraftingApp, SpreadsheetApp, etc.) and GUI automation are session-scoped. A process can only access COM objects created in its own Windows session.

The session trap: several approaches silently land the runner in Session 0 (background service context) where COM/GUI calls fail:

Verify the session before debugging COM errors:

import os, ctypes
session_id = ctypes.c_ulong(0)
ctypes.windll.kernel32.ProcessIdToSessionId(os.getpid(), ctypes.byref(session_id))
print(session_id.value)  # must be ≥ 1; 0 = wrong session

Fix: register the watchdog in the user's logon autostart via registry (correct session guaranteed):

reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\Run" ^
  /v EmergeRunner /t REG_SZ ^
  /d "wscript.exe C:\Users\<user>\.emerge\start_emerge_runner.vbs" /f

If runner is already running in Session 0, you must reboot (or log off/on) to get a clean Session 1 start — killing and restarting via SSH will land back in Session 0. After reboot the registry entry auto-starts the runner in the correct session.

The self-install script can create start_emerge_runner.vbs (Windows autostart). Manual equivalent:

Set sh = CreateObject("WScript.Shell")
sh.CurrentDirectory = "<plugin_root>"
sh.Run """<pythonw>"" ""<plugin_root>\scripts\runner_watchdog.py"" --host 0.0.0.0 --port <N>", 0, False

Linux/Mac — standard nohup launch

cd <plugin_root>
nohup python3 scripts/runner_watchdog.py --host 0.0.0.0 --port <N> \
  > ~/.emerge/watchdog.log 2>&1 &

No session isolation issues for headless Linux; use the install script or systemd user service from runner-install-url output.

ExecSession — Persistent Globals and COM Limitations

icc_exec calls to the same target_profile share one ExecSession (a persistent Python process). Globals survive across calls — like a Jupyter notebook cell:

# call 1 — import once
import numpy as np
data = np.array([1, 2, 3])
# call 2 — data and np are still in scope, no re-import needed
print(data.mean())

Exception: Windows COM objects — COM apartments are thread-local; COM objects do not survive across calls. See ~/.emerge/connectors/<vertical>/NOTES.md for vertical-specific patterns.

Non-COM globals (dicts, numpy arrays, plain objects) are safe to reuse across calls — skip redundant imports after the first call.

self-contained Import

remote_runner.py inserts its own root into sys.path — no PYTHONPATH needed regardless of working directory:

# scripts/remote_runner.py — top of file
_ROOT = Path(__file__).resolve().parents[1]
if str(_ROOT) not in sys.path:
    sys.path.insert(0, str(_ROOT))

Popup Push Flow

runner_notify delivers interactive popups daemon → runner without requiring the runner to be TCP-reachable (NAT-safe):

daemon.request_popup()
  → SSE push: data: {"type":"notify","popup_id":"<id>","ui_spec":{…}}\n\n
    → RunnerSSEClient._dispatch_command()  [runner-side thread]
      → _tk_dispatch(_build)              [queues to tk-main thread, blocks]
        → tk-main: Toplevel dialog shown, user interacts
      → on user action: on_result({"value":"…"}) fires
    → _post_result(): POST /runner/popup-result {"popup_id":…,"value":…}
  → daemon._on_popup_result(): ev.set()
daemon.request_popup() returns {"ok": true, "value": "…"}

toast (type=toast) is fire-and-forget: popup_id="", no result POST, daemon returns immediately.

Timeout layering (three levels, outer ≥ inner):

Windows Session requirement: operator_popup.py uses tkinter, which requires an interactive desktop session (Session 1). SSH-started processes run in Session 0 (no visible desktop) — popups will silently fail. Runner must be started via Registry Run key (installer sets this up) or by the user double-clicking the VBS shortcut.

Ops Endpoints

GET  /health        # {"ok": true, "status": "ready", "uptime_s": N}
GET  /status        # pid, python, root, uptime_s
GET  /logs?n=100    # last N lines of .runner.log
POST /run           # {"tool_name": "icc_exec", "arguments": {...}}
POST /operator-event                                      # append one event to local EventBus
GET  /operator-events?machine_id=&since_ms=&limit=        # read events since ts

Daemon endpoints consumed by runner (runner initiates, daemon receives):

GET  /runner/sse?runner_profile=<id>&machine_id=<id>   # SSE command stream (runner holds this open)
POST /runner/online                                     # runner registration heartbeat
POST /runner/event                                      # push operator events to daemon
POST /runner/popup-result                               # {"popup_id":"…","value":"…","attachments":[]}
POST /runner/upload                                     # file upload for rich-input popups

The runner accepts only `icc_exec` requests on /run. Pipeline bridge execution is handled by the daemon (icc_span_open when stable): it loads pipeline .py + .yaml locally, builds self-contained inline code, and sends it as icc_exec. Connector files never need to exist on the runner machine.

Request / response shape for /run:

{"tool_name": "icc_exec", "arguments": {"code": "...", "target_profile": "default", "no_replay": false, "result_var": "__emerge_pipeline_out"}}
{"ok": true,  "result": {"isError": false, "content": [{"type": "text", "text": "..."}], "result_var_value": {"rows": [], "verify": {"ok": true}}}}
{"ok": false, "error": "string message"}

EventBus endpoints

POST /operator-event — append a single operator event to the local EventBus file.

Request body (JSON):

{
  "ts_ms": 1712345678000,
  "machine_id": "workstation-01",
  "session_role": "operator",
  "event_type": "entity_added",
  "app": "desktop_drafting_app",
  "payload": {"layer": "annotation", "content": "master_bedroom"}
}

machine_id must be a plain identifier — no path separators or .. components. Response: {"ok": true} or HTTP 400 with {"ok": false, "error": "..."}.

GET /operator-events?machine_id=<id>&since_ms=<ts>&limit=<n> — read events newer than since_ms (default 0), up to limit (default 200, max 1000).

Response: {"ok": true, "events": [...]}.

These two endpoints are consumed by OperatorMonitor in the daemon when EMERGE_OPERATOR_MONITOR=1.

Common Mistakes