zby/commonplace/analyse-agentic-system

Analyse an Agentic System

Analyse one external agentic system at one frozen evidence boundary and publish its accepted artifact with the overview as the public entry point. Code runs the analysis as a workflow in one process: it opens the run, acquires the source, launches each role's worker through its harness's headless mode, checks and verifies every result, corrects what was refused, and publishes. You confirm the target, prepare the worktree, start the run, and report.

Invocation authorizes the run directory under kb/agentic-system-analyses/state/ and the retained analysis under kb/agentic-system-analyses/retained/<publication-name>/; code writes both. Code acquires and freezes a GitHub source's checkout under related-systems/; the boundary worker may freeze other sources under its source rules; later workers read them read-only. Invocation does not authorize editing source content, auxiliary indexes or surveys, transfer scans, landscape synthesis, other retained reports, or Git staging and commits.

Confirm the target

Before preparing a worktree or starting a run, identify the system and its source from the caller's input. If the target is missing or ambiguous, ask: “Which system should I evaluate? Please provide its repository URL or local checkout.” Stop and wait for the answer. Do not select a system from prior analyses or infer an unstated target from the current checkout. A supplied repository can identify the system without requiring the caller to repeat its name.

For a remote repository, apply this quick shape check to the trimmed locator:

^(?:https?://[^/\s?#]+/|ssh://[^/\s?#]+/|git@[^:\s]+:)[^/\s?#]+/[^/\s?#]+/?$

This accepts common HTTPS, SSH URL and SCP-style repository locators, including .git suffixes. Use the repository root, not an issue, file or documentation URL. If the locator does not match, ask for a repository locator or clarification; less common repository forms may be confirmed by the caller. A local checkout or explicitly supplied snapshot, bundle or documents uses its existing source route rather than this remote-URL check. Do not silently treat a malformed repository locator as a document source.

This is a plausibility check, not proof that cloning will succeed. Do not run git ls-remote, clone, fetch or test credentials during intake. Acquisition handles unavailable repositories, permissions and clone failures later.

The run uses a worker profile from worker-profiles.yaml: a harness, the model it launches and the effort. Every worker of the run uses it, verification included, and it is fixed for the run; another profile means another run. The advisory meta-analyser also uses that model and effort, but only on Pi, with ambient instructions, skills and extensions disabled. Use `pi-sol` (gpt-6.1-sol, medium effort) when the caller does not name a profile. The command defaults to it for new runs. The caller can override it, for example:

/analyse-agentic-system https://github.com/suzgunmirac/dynamic-cheatsheet
Use worker profile pi-ultramarine.

An explicit profile overrides the default. A resumed run always keeps its recorded profile; changing the default does not change existing runs.

Isolated run setup

Run each analysis in a dedicated Commonplace worktree at a committed method revision. Prepare it first, from the checkout this session started in:

commonplace-analysis prepare --name <system-label>

Use a nonempty label containing only lowercase letters, digits and hyphens, starting with a letter or digit ([a-z0-9][a-z0-9-]*). This preparation label is not the source-native display name: for example, use --name ace here and --system "ACE" when starting the run.

Code creates a detached worktree under .commonplace/worktrees/, installs its local commands from the committed lockfile, verifies their binding, and prints a JSON record. Keep its worktree and path-prefix values. --worktree <new-path> overrides the location. Existing destinations are refused. Failed installations retain the worktree and a sibling .preparation.json record for diagnosis. Preparation writes Git metadata and may download packages: when the sandbox blocks it, request escalation for this command alone. For a label-only rejection, you may correct only --name and retry without operator escalation, but only before any worktree or run has been created and with the target, source and scope unchanged. For every other preparation failure, stop and give the operator its message; do not retry automatically. Do not start a run in the originating checkout.

Preparation uses HEAD. It stops when HEAD is behind the default branch. Stop and tell the operator; only the operator selects another method with --revision <commit>.

By default, preparation permits unrelated uncommitted changes in the originating checkout, excludes them from the new worktree, and records their exclusion in its JSON record. --require-clean-origin instead stops preparation when the origin is dirty. Startup instructions and configuration must match the selected commit in either mode: instruction files such as AGENTS.md, project harness settings and system prompts, skill and agent directories, extensions, hooks and prompt resources, and repository-local targets of skill symlinks. Staged, unstaged, untracked and ignored startup files are checked, except an ignored settings.local.json, which holds one user's harness permissions.

Continue in this session:

• Run every later command of this skill with worktree as its working

directory, calling it as <path-prefix>/commonplace-analysis.

• Every path this skill names is relative to worktree.

Code refuses a run's command when it is called from another checkout or runs another checkout's code, and the refusal names the directory to use. Follow it; do not reinstall the shared user-level tool to point at the run.

Keep the worktree's code, method files, lockfile and environment unchanged until the run finishes; develop and merge method changes elsewhere. A completed published run stays in this worktree until the operator separately authorizes integration.

A command after -- instead runs in the prepared worktree with its commands on PATH; this session then stops after preparation. The operator may use it to start a harness there, or to run the analysis in one step:

commonplace-analysis prepare --name <system-label> -- \
  commonplace-analysis run --system "<name>" --source-identity "<identity>" \
    --source "<input>" --worker-profile <profile>

Worktree removal is a separate operator decision. It is permitted only when every run in it has completed with verified publication merged into main, no integration branch awaits a merge or conflict decision, and every audit has extracted its evidence record or the operator has said none needs it. Then use git worktree remove, remove the sibling preparation record, and delete merged analysis branches. Never remove a worktree containing a stopped run through this routine: failures, rejected outputs and traces are evidence. Completed local blocked/out-of-scope results also need a separate disposition decision. Removal is never automatic.

1. Run the analysis

Start it:

commonplace-analysis run \
  --system "<source-native system name>" \
  --source-identity "<stable source identity, e.g. https://github.com/owner/repo>" \
  --source "<the caller's source input, as given>" \
  --worker-profile <profile> [--source-revision <full 40-hex commit>]

Code normalizes the source identity, allocates AAS-<date>-<source-slug>-<worktree-token>-<nn> under kb/agentic-system-analyses/state/, prints that path, and runs the workflow there. The token comes from the ready preparation record. The run's source slug is independent of its publication name: GitHub sources publish under lowercase <owner>--<project>, preserving repository punctuation; other sources keep their existing naming. There is no public-path override.

For a GitHub source, add --source-revision when the caller requests a revision or asks to keep the existing checkout's commit; for the latter, read git -C <checkout> rev-parse HEAD and pass that full commit. The option requires a GitHub identity. Code freezes related-systems/<owner>--<repo>/ before the boundary worker: without a revision it clones a missing checkout or fetches a clean existing one and detaches it at the default branch tip; with a revision it uses that commit. A dirty checkout, foreign origin or unavailable commit stops the run for the operator. If the sandbox is known to block the network, request approval for the run command before running it; do not clone or fetch yourself.

The run opens the analysis and pins the prepared method commit, then runs the roles in the workflow's order: boundary; runtime; memory, epistemic and workflow in parallel; reconciliation; report verification with its correction rounds; memory profile and synthesis, each verified; assembly and publication. Each worker is a fresh headless session of the profile's harness in the worktree, with up to --max-workers (default 3) running at once and --timeout seconds (default 10800) per launch. The run holds a lock: a second run on the same run directory is refused.

A run takes hours. Start it so that it outlives your tool call (a background command whose exit you wait for, or a terminal the operator keeps open), wait for it to exit, and read its exit status and output. Exit status 0 means the run completed: it printed the publication receipt, with published: true for a complete disposition or published: false for a blocked or out-of-scope boundary, which completes locally. Exit status 2 means the run stopped and printed why; 1 means the command itself failed.

Advisory trace audit

Before run exits, code saves a scan of every manifest-recorded attempt, including failed and superseded ones. The scanner detects explicit Pi tool failures; missing, incomplete and unsupported traces remain visible as coverage limits. It does not discover nested workers outside the manifest.

After a completed Pi analysis, code passes the scan, attempt inventory and accepted artifact paths to a fresh meta-analyser. It uses failures as leads for at most three verification questions and 30 tool calls, with a timeout of 900 seconds or --timeout, whichever is shorter. The question and call limits are prompt instructions, not a tool sandbox. Only read and bash tools are exposed; read-only authority is prompt-enforced. Code saves the returned text. No findings is not a correctness certificate.

<run>/audit/status.json identifies the latest snapshot. Each audit/<input-digest>/ keeps scanner.json, attempts.json, the input digests, exact prompt, launch record, stdout, stderr, status and any completed report. An unchanged rerun reuses its snapshot, including a failed audit; changed inputs create a new one. Audit failure, timeout or incomplete coverage does not change the analysis receipt or exit status. A stopped analysis is scanned but does not launch the meta-analyser. Other harnesses do not launch it.

The audit is advisory: report material findings and unresolved concerns to the operator. Do not repair accepted artifacts or rerun workers on its authority. Audit evidence is local run state, not included by integrate. Before authorized worktree removal, extract useful audit evidence to a retained destination or obtain the operator's explicit decision that none needs retention. After an abrupt controller death, inspect the audit's recorded PID and confirm its process and children have stopped; an interrupted snapshot is not automatically retried.

2. Stops and reruns

Every worker answer is recorded under its full request, so

commonplace-analysis run <run>