brianmoney/opsx-controller/opsx-plan-ops

opsx-plan-ops — operations playbook

opsx-plan drives OpenSpec changes through implement → review → archive using model workers. The controller is deterministic; the workers are not. Nearly every plan failure is one of a small set of known shapes — triage by symptom first, root-cause second, and fix at the correct layer (controller repo, model config, plan content, or operator process).

Mental model

• A plan (a TOML manifest, conventionally openspec/plans/<name>.toml)

lists changes in phases with depends_on, plus gates: require_clean_tracked, fast_checks, pause_before, escalate_after_review_fails, invalid_output_retries.

• Each change loops rounds of implement → review; a zero-finding

review promotes to archive. pause_before = true changes wait for opsx-plan approve <change-id>.

• State lives in .opsx-plan/ under the target repo: <plan>.state.json

(controller state), workers/<plan>/<change>.json (per-change worker state), logs/<change>.<stage>.r<round>.<attempt>.log (one file per worker invocation).

• Worker output contract: every worker's final message must be exactly one

line containing one JSON object — no prose before or after. The controller parses the log for that line; a worker that ends in prose fails the stage with output invalid: expected a final JSON object line. invalid_output_retries (default 2) re-runs the stage with a RETRY_CORRECTION hint before failing the change.

• Unattended runs are owned by the opsx-plan autopilot wrapper, a

systemd user unit — not by a foreground shell. Autopilot loops opsx-plan run, auto-resets transient failures, gates pause_before changes behind a veto window, and escalates everything else. See docs/opsx-plan-operator-workflow.md → "Autopilot (unattended runs)".

Pre-flight (before every opsx-plan run)

1. git status --short — require_clean_tracked = true (the default) refuses

a dirty tracked tree. Workers commit wip: <change-id> commits as they go, but a killed run often leaves finished work uncommitted. Commit it (same wip: style) before launching; ask the owner before any git mutation.

1. opsx-plan status — know which changes are pending/failed/blocked and

which phase a failed change died in.

1. opsx-plan doctor — preflight checks: models resolve, client on PATH,

tree clean, plan loads. Cheaper than discovering these mid-run.

1. Confirm the worker models and variants are sane:

opsx-plan models show --adapter <adapter>. Variants are model-specific effort labels; an invalid variant is silently dropped to default (see triage table).

Launch discipline

For the full step-by-step operator procedure, see docs/opsx-autopilot-runbook.md.

opsx-plan run is a foreground, serial process: if it is a background job of an agent shell, a shell timeout or cleanup kills the whole process group — including the controller mid-stage. For anything past a quick --max-changes 1 smoke run, do not hand-roll setsid nohup. Launch unattended work through the autopilot wrapper, which supervises opsx-plan run in a loop from a systemd user unit.

# One-time: bind the unit to this repo + plan and enable it. The install
# command renders the installed unit template and plan drop-in (WorkingDirectory,
# OPSX_PLAN, and a derived toolchain PATH) under ~/.config/systemd/user/, runs
# daemon-reload, enables, and never starts. --print previews without writing.
opsx-plan autopilot install --plan openspec/plans/my-plan.toml
systemctl --user start  opsx-autopilot
 
# Watch it
systemctl --user status opsx-autopilot
journalctl --user -u opsx-autopilot -f

What autopilot does in your stead (full detail in docs/opsx-plan-operator-workflow.md → "Autopilot (unattended runs)"):

• Transient failures (subagent_output_invalid, timeouts) get a bounded

auto-reset — default 2 per failure signature, at least 300s apart. Signatures persist in .opsx-plan/autopilot-state.json, so they survive opsx-plan reset.

• `pause_before` gates: autopilot notifies, waits a veto window (default

30 min), then auto-approves. Veto by creating .opsx-plan/veto/<change-id> during the window; approving manually short-circuits the wait.

• Acceptance gates: an orchestrator-created change (review_created = true)

is announced (opsx-plan accept: <change-id>) and autopilot waits — no auto-accept — until opsx-plan accept <change-id> records your acceptance; it then resumes by itself.

• Permanent classes (billing/quota, permission rejections,

finding_recurrence_exceeded, max_rounds_reached, no_progress, archive failures, unknown) escalate immediately — never auto-retried.

• Deterministic environment failures (dirty tracked tree / uncommitted

archive output, unresolvable plan, missing opsx-plan executable) write .opsx-plan/autopilot-paused.json, escalate once, and exit 0. Later starts, including explicit unit starts and --once, no-op before plan resolution with only a paused event. Only known deterministic reasons pause; execution-lock contention and unclassified engine exits 2 stay transient.

Escalation appends a digest to .opsx-plan/escalations.jsonl (change id, class, last_result, reason, finding loci, attempt count, stage log path, suggested action), sends an ntfy push, and leaves the unit down until you fix and restart it (change-level classes and deterministic environment pauses exit 0; transient environment exits 2 and is retried until StartLimitBurst trips). opsx-plan autopilot status [--plan X] is a read-only pause and recorded plan/change-state snapshot, even with a broken plan. A durable pause must be cleared with opsx-plan autopilot resume [--plan X]: it rechecks plan resolution, the clean-tracked requirement, and the executable; failure exits 2 with the marker intact, success clears it and exits 0 without starting the unit. Use the unit's repo/plan/PATH; changing plans or resetting systemd is not a bypass. Every decision is logged to .opsx-plan/autopilot-events.jsonl. Monitor with opsx-plan status and the journal; worker logs still stream into .opsx-plan/logs/, and opsx-plan logs --follow selects the in-progress one.

Triage: symptom → cause → action

Autopilot's failure classifier mirrors this table: the transient rows (worker output contract misses and provider errors) get a bounded auto-reset; everything else escalates rather than being retried.

Trusted model + deterministic context (for semantic fixes)

When the blocker is a judgment call rather than a mechanical bug (spec identity, requirement renames, contract interpretation), do not let the incident-handling agent improvise it inline, and do not hand it to a cheap worker. Dispatch one trusted model with a fully bounded context:

1. Name the exact files to read (delta vs canonical, not "the specs").
2. State the failure message and the governing semantics (e.g. "a

requirement's name is its identity; MODIFIED must match verbatim").

1. Enumerate the acceptable options and the selection criterion (minimal

churn, identity preserved).

1. Constrain scope: which file(s) may be edited, which validations must run,

no commits.

1. Require a one-paragraph decision report as the final message.

Invoke it as a one-shot with your client's non-interactive mode and an explicit strong model. Verify the diff afterward — trust, but read the git status.

Recovery playbook

1. Fix the root cause at its layer (see table).
2. Commit any uncommitted worker output — the run that follows a reset

refuses a dirty tracked tree (require_clean_tracked).

1. opsx-plan reset <change-id> — returns the change to pending so it

re-enters the implement phase. A reset change with all tasks complete re-runs implement as a fast no-op, then review, then archive. This is normal, not a loop.

1. opsx-plan autopilot status [--plan X] — if an environment pause is present,

run opsx-plan autopilot resume [--plan X] with the unit's repo/plan/PATH after fixing it. Do not delete the marker to bypass preflight. If transient retries hit systemd's start limit, also systemctl --user reset-failed opsx-autopilot.

1. Restart autopilot and watch the first stage transition before walking away:

systemctl --user start opsx-autopilot. (An unattended run that escalated left the unit down on purpose — autopilot never retries a permanent class; see the "Launch discipline" section.)

1. After done, the plan stops at the next pause_before change — that is

the checkpoint to review diffs before approving.

Forensics: adapter session records

When the worker log isn't enough (did the model stop, crash, or get truncated?), the adapter client usually keeps its own session records. One worked example, for the opencode adapter: