burtenshaw/multiautoresearch/autolab-managed-experiment

Use this for any single Autolab experiment that should result in exactly one managed benchmark run.

Workflow

1. Load the local operator env and refresh from local master:

• . ~/.autolab/credentials
• uv run scripts/refresh_master.py --fetch-dag

1. Edit only train.py for the single intended hypothesis.
2. Run preflight before launch:

• uv run scripts/hf_job.py preflight

1. Launch exactly one managed experiment:

• uv run scripts/hf_job.py launch --mode experiment

1. Stream logs and parse the final metric:

• uv run scripts/hf_job.py logs <JOB_ID> --follow --output /tmp/autolab-run.log
• uv run scripts/parse_metric.py /tmp/autolab-run.log

1. Record the run locally:

• uv run scripts/submit_patch.py --comment "..."

1. Promotion is local and only happens if val_bpb beats current master.

Guardrails

• Treat train_orig.py as the refreshed local-master base. If preflight reports

multiple known hypothesis categories, stop and inspect the diff before launching.

• Ignore repo git main and origin/main when judging freshness. In this rig

repo those refs describe control-plane history, not the benchmark master. The comparable base is whatever refresh_master.py just wrote into train_orig.py, research/live/master.json, and research/results.tsv.

• Never run uv run scripts/hf_job.py launch --mode prepare from an

experiment-scoped worktree. prepare is shared bootstrap work, not per-experiment work.

• Do not launch a second experiment job for the same experiment unless you have a

specific reason and intentionally override the duplicate check.

• If the workspace looks stale against the current local master, stop and rewrite

the experiment rather than rationalizing the mismatch.

Fast Checks

• uv run scripts/hf_job.py preflight --json

Use this when you need to inspect the diff preview, active conflicts, or detected change categories programmatically.

• uv run scripts/trackio_reporter.py summary --max-jobs 25

Use this to confirm the experiment id or hypothesis is not already active.