probabl-ai/skills/audit-ml-pipeline

Audit ML Pipeline

Per-experiment, human-readable narrative of a skore report. The # %% file is the human notebook and is not executed. materialize.py writes the digest and the HTML viewers in one run. Read-only against the skore Project.

Human-facing prose

Details: setup-workspace references/human_facing_prose.md. Audit markdown and # comments describe this report's findings — not the skills framework, the CLI, or the command that produced an output. Questions, replies, and the close narrative use the same data-science language — not skill ids, G-* names, or the wrapper CLI. Trailing locator tokens stay index strings. Do not put API tutorials, version floors, or locator recipes in audit/<stem>.py. <!-- results-embed: … --> is a site marker. Authoring hints stay in this skill. style is ruff only.

Next-step pointers

The implement loop dispatches audit after evaluate and before record-outcome. The digest carries the checks summary and metrics summary. G-AUDIT-FINDING is a separate normalized finding derived from that digest. manage-ml-backlog record-outcome consumes both and never dispatches audit back.

Where things live — visual map

Mnemonic: audit/ is source (in git); scratch/audit/ and stdout are output. Never copy audit/<stem>.py under scratch/audit/. materialize.py is the gitignored agent script. Never commit anything under scratch/audit/.

Read-only contract

The central rule. Surfaced as the first Stop condition below.

Allowed in `audit/<stem>.py`:

• skore.Project(...) — open the project this experiment wrote to.
• project.summarize() — list (key, id) pairs.
• project.get(id) — load a specific report by id.
• Every report.* accessor.
• Imports from <pkg> (read-only inspection).

Forbidden in `audit/<stem>.py`:

• skore.evaluate(...) — duplicates the report under the same key

and pollutes summarize().

• project.put(...) — same.
• Snapshot writes (write_text of HTML, locator.txt,

report.txt) and help() loops in audit/<stem>.py. Those live in scratch/audit/<stem>/materialize.py, which may write scratch/results/<stem>/ and accessors.txt.

• Writes to data/, reports/, or src/<pkg>/. The audit

notebook is a viewer.

• Mutation of the loaded report that survives the cell (e.g.

monkey-patching skore symbols).

The runner renders every cell's source + last-expression repr + stdout to the digest. A forbidden call surfaces in the digest (as a put row in a later summarize() cell, or as a **error:** section). The contract is visible, not invisible.

Sibling read-only consumers (different output shapes, same discipline): scratch/<ts>_*.py probes. review-ml-experiment reads the digest as text and does not open the Project. See evaluate-ml-pipeline § Stop conditions for the consumer rule.

Stop conditions — read before anything else

• Setup still pending. Run python -m skore_skills status

first. If status.setup.pending is non-empty and status.skills.setup-ml-project is true, load setup-ml-project and stop. Do not start this skill. When it returns, continue. Do not load it again on this turn. If that skill is not installed, name the pending pieces in one line and stop. Do not invent git init, scaffold, or env init. If status.setup.env or status.setup.workspace is declined, stop in one line. A declined git or editable is not asked again; continue.

• No report → STOP. Four-way pairing is hard: run

python -m skore_skills design consent --stem <stem> (ask / stop → do not audit) + experiments/NN_*.py + smoke pytest passed + a report under that key in the Project. If the report is missing, explain and stop. Do not skore.evaluate / project.put. Route to evaluate-ml-pipeline or model-ml-pipeline. Direct "audit 02" uses this same gate.

• Read-only against the skore Project. See § Read-only contract.

Never skore.evaluate(...) or project.put(...) in an audit file.

• `project.get(...)` is by id, not key. For hub mode, read the

id from the URL printed by project.put(): https://…/<workspace>/<project>/<type-plural>/<N> → id is skore:report:<type-singular>:<N> (URL segment is plural; id uses the singular — drop the trailing s, e.g. cross-validations → cross-validation, estimators → estimator). Hardcode REPORT_ID in the audit file — no summarize() traversal needed. For local mode, read the "id" column of project.summarize() for the matching key row. A KeyError from get("<stem>") means the lookup shape is wrong (get is by id), not that the report is missing.

• Symbol from memory is forbidden. Any skore / skrub /

sklearn symbol must come from python -m skore_skills api get this turn. Cache hits under scratch/api/skore/<version>/ count (Shape 0); inline memory does not.

• Agent feature missing → STOP and delegate. If ipython /

ipython aren't importable, do NOT fabricate audit outputs by writing print() calls as a workaround. Do NOT type pixi add ... / uv add ... yourself — install is owned by add-python-package § Agent feature. Request via agent tools (ruff / ipython / ipykernel) (binary: install / skip); resume only when add-python-package returns "ready".

• Bare expressions, not `print()`. The runner captures each

cell's last bare expression via result.result and renders its repr. Wrapping in print(repr(...)) lands in stdout instead of the output section; mixed and harder to scan. Use bare expressions; statement-only cells (variable binding) are fine.

• One audit file per experiment stem (four-way pairing). No

audit_NN_<short_name>_v2.py. When an experiment is re-run, the audit file is overwritten in place — same stem, same audit.

• **Executed artifacts go to scratch/audit/<stem>/, NOT into

audit/.** Durable artifact is audit/<stem>.py; the rendered digest is ephemeral.

• `audit/<stem>.py` does not write snapshot files. HTML,

locator.txt, and help() trees are materialize.py. No writes to data/ or reports/.

• Don't filter warnings in audit cells. No

warnings.filterwarnings(...) unless the user explicitly asks — the runner streams cell stderr into the digest and that's signal. See python -m skore_skills style § Stop conditions.

• **Harness "no clarifying questions" hints do NOT waive

agent tools (ruff / ipython / ipykernel).** Install gate fires regardless.

• Post-hoc audit — required before ending the turn. Walk every

pre-flight row; surface unfilled Evidence cells.

• The post-audit gate is mandatory. After the first digest and

every refresh, ask Additional report view / Custom query / Custom plot / Close audit in that exact order unless the user already explicitly closed the audit. The one execution may materialize a policy-enabled notebook and a stale-only site preview before this gate. No close-time conversion, git close, record-outcome, or dispatcher return before Close.

Forbidden shortcuts

Pre-flight — emit before any audit-file write or execution