xia-ataraxia/craft-skills/ml

ml

Run ML/DL research and engineering work under one discipline: reproducibility first, evaluation honesty second, novelty third. Label preliminary or non-reproducible observations honestly; do not present test-tuned numbers as independent evidence or claim an improvement without a comparable baseline. This skill is an index — shared rules live here, per-topic methods live in references/; load the matching reference before touching a project layout, a dataset, or a training run.

Output contract

Produce the requested modeling artifact or review with the relevant dataset, configuration, source identity, observed outcomes, and verification limits. Preserve the incumbent environment and distinguish intended commands from actual runs. Missing launch authority, unresolved input identity, or a failed required recovery test blocks that launch, not unrelated read-only diagnosis. Report missing evidence explicitly; never invent a metric, successful run, or reproducibility receipt.

Task gate — run first, every time

Identify the task type before writing dataset, training, or vision code. Rows stack — load every reference whose row matches, not only the first one that fits:

Example: labeling and splitting an image dataset that will then be trained on matches three rows at once — load datasets.md + training.md + vision.md together, not vision.md alone.

Core rules

• Reproducibility receipt. Bind a compared result to its source revision and frozen current-content identity, configuration, data manifest, actual invocation, and observed outcomes. Qualify missing evidence rather than inventing it. references/training.md owns the full tracking contract, including reported generative evaluations.
• Split before you fit anything. Train/validation/test separate before any statistic is computed from the data — scaling, imputation, vocabulary, augmentation parameters. references/datasets.md gives the three leakage classes and their detection commands.
• A baseline exists before a novel approach is judged. "Better than the majority-class/linear/frozen-pretrained baseline," not "better than nothing." references/training.md covers the discipline ladder in full.
• Reserve test data for frozen final evaluation, never model selection or tuning; disclose repeated evaluation rather than calling a reused test set fresh evidence.
• Mutable library/runtime facts. For framework APIs, installation, compatibility, or runtime behavior, consult the library's official documentation first. Disclose conflicts; a more-specific local contract or reproducible evidence for the matching library version and platform may override general or stale documentation. If unresolved, leave it unknown and stop or use the applicable safe fallback — never invent a capability or command.

Requirements

• Python: for a greenfield project, uv is a default for dependency locking and running; pandas/polars, pandera (or an equivalent schema checker), and scikit-learn/torch/jax are examples chosen to fit the task. In an established project, use its existing locked environment and installed tooling.
• Runtime and framework maintenance: record the applicable official Python documentation, python3 --version, the incumbent lockfile, the installed framework version from its documented in-environment probe, and the selected framework's official release source. When probe or release evidence shows a selected runtime or framework's runtime form changed, recheck the official documentation, rerun affected package evaluations for datasets, training, and GPU work, update the recipe if needed, then bump this package's version and record the reason in the commit message before reporting results.
• git, grep, find, awk, sha256sum (shasum -a 256 on macOS) for the detection commands in each reference.
• A config format the team already uses for one-run-one-config (examples assume YAML).

Common rationalizations

Red flags

• A greenfield project with no locked environment or importable training code, or an unrelated feature that rewrites an established project's packaging.
• A preprocessing/scaling/vocabulary step that runs before the split, or on the concatenation of all three.
• A "novel" result reported with no baseline number in the same table.
• A compared run whose actual code/configuration inputs cannot be recovered from its recorded revision and frozen snapshot.
• A robust-improvement claim without uncertainty evidence appropriate to the task, or invented variance for a single run.
• An augmentation transform present in the validation or eval data loader.

Boundaries

Not for wrapping a trained model behind a serving API — load principle-backend — or for suite-level test-architecture decisions — load principle-testing. GPU/CUDA environment, compatibility, and shared-host launch preflight are gpu's domain — a GPU training run loads gpu first, then this skill. Double-check the agent-behavior boundary from the task gate: "the model calls a tool" or "the pipeline reasons over retrieved text" is agent work even when it also touches a model file.

Verification

• [ ] The task gate identified the task type and the matching reference was read before writing code — or the task was recognized as agent work outside this package.
• [ ] The established project's locked environment and layout were preserved, or greenfield code has a locked environment and is importable.
• [ ] Every fitted statistic (scaler, vocabulary, augmentation parameter) is fit on the train split only.
• [ ] A baseline number exists in the same report as any novel-approach number.
• [ ] Each reported result has a reproducibility receipt: the code revision, configuration, data manifest, and recorded outcome identify what ran.
• [ ] Test data did not drive tuning or model selection, and any frozen evaluation replay is disclosed.
• [ ] Claims report the actual sampling/repeat count and appropriate uncertainty; limited evidence is labeled without a universal seed quota.