nvidia-tao/tao-skill-bank/tao-generate-anomalies

TAO Generate Anomalies

Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).

Multi-phase pipeline (0–7); the mode flag selects which phases run.

Run every phase through to completion without mid-run pauses. Collect all required parameters up front, and run every command from the repo root.

Shell setup. All ${ANOMALYGEN_SCRIPTS} references resolve to the packaged helper-script directory. Inside the container this is preset (ENV ANOMALYGEN_SCRIPTS=<dir>/scripts/utilities); on the host, export it once per shell:

export ANOMALYGEN_SCRIPTS="$(git rev-parse --show-toplevel)/scripts/utilities"

python3 -m scripts.utilities.<name> invocations work from any CWD inside the container (PYTHONPATH is preset) and from the repo root on the host. When inside a product container (ANOMALYGEN_PRODUCT_MODE=1), invoke anomalygen-guard before any GPU work; if it reports BLOCKED, fix the listed issues before continuing.

Quick Start

The pipeline runs inside the metropolis_sdg.paidf_anomalygen container (declared in versions.yaml) or any host with the cosmos-predict2 conda env active. All phase commands assume that environment, at the repo root, with ANOMALYGEN_SCRIPTS exported.

Minimal end-to-end run (mode=full):

# 1. Set the shared variables (see "Shared variables" for the full set).
export ANOMALYGEN_SCRIPTS="$(git rev-parse --show-toplevel)/scripts/utilities"
MODE=full
NAME=my_exp
DATASET_DIR=/data/uc1
DEFECT_DESC=assets/defect_spec_template.jsonl
NUM_SDG=20
MODEL_SIZE=2b
 
# 2. Phase 0 — verify / download checkpoints (~40 GB for the 2B default; needs HF_TOKEN).
${ANOMALYGEN_SCRIPTS}/check.sh --model-sizes ${MODEL_SIZE^^} \
    || ${ANOMALYGEN_SCRIPTS}/download_checkpoints.sh --model-sizes ${MODEL_SIZE^^}
 
# 3. Walk Phases 1→7 in order (see each Phase section).

For mode=inference_only (reuse a checkpoint) also set CKPT/STEP and skip Phase 1. For mode=finetune_only run only Phases 0–1.

Running in Docker — container launch, mounts & permissions

The paidf-anomalygen image runs as a non-root baked-in user (USER anomalygen, uid=10000), independent of your host uid. Docker does not remap uids on bind mounts, so a host directory owned by your uid is not writable by uid 10000 and the container fails the instant it tries to create a file there. Run with the full host identity: --user "$(id -u):$(id -g)", matching USER/LOGNAME, HOME=/tmp, and read-only /etc/passwd+/etc/group mounts, plus the cache redirects. Run the fail-fast write preflight before Phase 0. See references/docker.md for the full docker run command, the load-bearing-flag table, the preflight snippet, and the uid-10000 chown/chmod fallback.

Reference files — read before executing phases

Read `references/finetune.md` before Phase 0/1 and `references/inference.md` before any of Phases 2–7; for mode=full read both before starting. The remaining references below are on-demand — read when troubleshooting or needing full detail for a specific phase.

Required parameters

num_SDG allocation depends on prep_testcase.sh --mode: inference (default, Phase 2) is uniform across defect types, override per-defect via --per-defect-counts; validation (Phase 1's validation JSONL) is proportional to training mask counts (largest-remainder rounding) and enforces ≥1 per defect. See references/prep-testcase.md for the full mode table.

Conditionally required

Optional parameters

Mode validation (fail fast before any phase)

• mode unset → halt: "`mode` is required (`full` | `inference_only` | `finetune_only`)."
• mode=inference_only missing either checkpoint_dir or step → halt: "inference_only requires both `checkpoint_dir` and `step`."
• mode=full with checkpoint_dir or step supplied → halt: "full mode runs finetune; use `mode=inference_only` to reuse an existing checkpoint."

Shared variables

Set once before Phase 0:

MODE=<full|inference_only|finetune_only>
NAME=<exp>
DATASET_DIR=<dataset_dir>
CLEAN_DIR=${clean_dir:-${DATASET_DIR}}
CKPT=<checkpoint_dir>      # required iff MODE=inference_only; auto-derived after Phase 1 when MODE=full
STEP=<iter>                # required iff MODE=inference_only; auto-derived after Phase 1 when MODE=full
NUM_SDG=<N>
DEFECT_DESC=<defect_spec.jsonl>
DEFECTS=(T+A T+B)          # TEXTURE+TYPE names. For mode=inference_only, derive from ${CKPT}/ag_config.yaml → dataloader_train.dataset.anomaly_types (also printed by validate_checkpoint.py in Phase 0). For mode=full, take from DEFECT_DESC entries. See references/inference.md §Phase 0.
NUM_SEARCH_RUN=${num_search_run:-3}
NN_THRESHOLD=${nn_threshold:-0.4}
MODEL_SIZE=<2b|14b>
NUM_GPUS=${num_gpus:-1}
MAX_ITER=${max_iter:-75000}
SAVE_ITER=${save_iter:-5000}
VALIDATION_ITER=${validation_iter:-5000}
LR=${lr:-0.02}
BATCH_SIZE=${batch_size:-2}
IMAGE_SIZE=${image_size:-512}
VALIDATION_JSONL=${validation_jsonl:-}  # optional; set by Phase 1 Step 2 if not user-supplied
 
BASE=results/${NAME}
JSONL=ag_inference/${NAME}/testcase.jsonl
ORIGINAL=${BASE}/original
SEARCHED=${BASE}/searched
ROUNDS=${BASE}/rounds
REGENS=${BASE}/regens

Guard preflight (product mode only)

When ANOMALYGEN_PRODUCT_MODE=1, run .agents/skills/anomalygen-guard/scripts/preflight.py before any GPU work and fix any BLOCKED issues. --validation-jsonl is forwarded only when the user supplied one; for MODE=finetune_only omit --num-sdg if not supplied. See references/guard-and-custom-counts.md for the full preflight command with all forwarded flags and the validation-JSONL / allocate_samples.py 0-entry checks.

Phase 0 — checkpoints

Read references/finetune.md §Phase 0 for HF_TOKEN requirements and what gets downloaded. Both scripts default to the 2B base + t5-large (~40 GB); pass --model-sizes ${MODEL_SIZE^^} so the chain checks and fetches the base size this run actually uses (2b→2B, 14b→14B) — otherwise a 14b run silently passes the 2B-only check and never downloads its checkpoint. Verify first; download only what is missing.

${ANOMALYGEN_SCRIPTS}/check.sh --model-sizes ${MODEL_SIZE^^} \
    || ${ANOMALYGEN_SCRIPTS}/download_checkpoints.sh --model-sizes ${MODEL_SIZE^^}

Phase 1 — fine-tune (skip when MODE=inference_only)