nvidia-tao/tao-skill-bank/tao-analyze-gaps-visual-changenet

TAO VCN Classify Gap Analysis Skill

Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).

You are an analyst for NVIDIA TAO VCN Classify (Visual Component Net) inference results. Your job is to identify the weakest samples per ground-truth label by measuring signed distance from the decision threshold in the wrong direction, then surface them for downstream augmentation or relabeling.

This skill is intentionally lightweight. VCN's classify head is a single-score binary boundary (PASS vs NO_PASS by siamese_score), so the analysis is computational, not investigative. The whole computation lives behind one direct docker run invocation against the pinned TAO data-services image (see Setup). The container's entrypoint takes <category> <action> [hydra overrides...]; we pass gap_analysis vcn_aoi key=value …. Each override is a bare Hydra key=value that selectively overrides the script's GapAnalysisConfig schema (defaults are baked into the container; use the mounted minimal-spec --cfg=job recipe immediately below to introspect them). (There is no dataset keyword inside the container — that's the TAO launcher's pillar prefix and is dropped here.) You do not need delegated analysis, multi-phase image audits, or component-type clustering — VCN does not expose those dimensions. View only a small set of representative weak samples to qualify the gaps after the container returns.

CLI surface can shift between data-services container builds. If a gap_analysis vcn_aoi invocation fails on argument parsing, introspect the actual schema once per image with a minimal spec on a bind-mounted host path:

SPEC_DIR="$(mktemp -d)"
printf '%s\n' 'min_recall: 0.99' 'top_k_per_label: 5' > "$SPEC_DIR/vcn_aoi_spec.yaml"
docker run --rm -v "$SPEC_DIR:/w:ro" "$DS_IMAGE" gap_analysis vcn_aoi \
    -e /w/vcn_aoi_spec.yaml --cfg=job

The -e path must resolve inside the container, so the spec must live under the directory mounted at /w. Reconcile any renamed keys (e.g. inference_csv vs inference_results_dir, output_dir vs results_dir) before retrying. Output parquet name is kpi_gaps.parquet.

Inputs

1. Experiment result directory — contains inference/inference.csv from TAO VCN Classify inference. Required columns: input_path, object_name, label, siamese_score. Pass the directory (e.g. inference/latest/), not the CSV file — the container reads inference_results_dir/inference.csv.
2. Training code/config directory — contains the VCN train YAML. The container reads dataset.classify.input_map (lighting condition list) and dataset.classify.image_ext from it to expand each weak sample into one row per lighting.
3. Dataset directory — image root prepended to the relative input_path from each row (kpi_media_path).
4. Schema overrides — min_recall, top_k_per_label, and optionally a hard-pinned threshold are passed as Hydra overrides (defaults: min_recall=1.0, top_k_per_label=50, threshold=-1.0 meaning sweep). `top_k_per_label` must be a positive integer — omitting it flips the container into "below-threshold filter" mode, which at min_recall=1.0 returns only PASS misclassifications and zero NO_PASS rows. See Common pitfalls.

Setup

The threshold sweep, weakness ranking, and per-lighting expansion all run inside the pinned TAO data-services image below. Confirm Docker, the NVIDIA container toolkit, and a GPU are present and ensure the image is cached:

# Resolve the pinned TAO data-services container URI from the release manifest.
DS_IMAGE="$(scripts/resolve_versions_key.py images.tao_toolkit.data_services)"
echo "DS_IMAGE=$DS_IMAGE"
 
docker info > /dev/null && echo "OK: docker"
nvidia-smi > /dev/null && echo "OK: GPU"
docker image inspect "$DS_IMAGE" > /dev/null \
  || docker pull "$DS_IMAGE"

A GPU is required; aborting early on a GPU-less host saves a confusing late error.

Three setup rules are load-bearing and easy to get wrong:

• Path mounting — every host path the container reads or writes (inference.csv, train YAML, dataset image root, output dir) must be bind-mounted, simplest with -v $WORKSPACE:$WORKSPACE -w $WORKSPACE so absolute paths resolve identically on both sides.
• Do not pass `--user $(id -u):$(id -g)` — it triggers KeyError: 'getpwuid(): uid not found: <uid>' during the container's transformers import; chown outputs back to the host UID afterwards instead.
• `-e <spec>` is required, not optional — current images hard-require it and exit with ValueError: The subtask vcn_aoi requires the following argument: -e/--experiment_spec_file before parsing CLI overrides.

See references/container-setup.md for the full path-mounting pattern, the --user/chown rationale and alpine chown command, multi--v guidance, and the -e <spec> requirement detail.

Method

The whole skill is a single docker run invocation followed by a small visual spot-check. The container does Steps 1–4 internally (threshold sweep, weakness scoring, top-K selection, per-lighting expansion). You handle Step 5 (visual spot-check) directly with the Read tool.

Step 1–4 — Run the container

$DOCKER gap_analysis vcn_aoi \
    inference_results_dir=<exp_dir>/inference/<label>/ \
    train_config=<exp_dir>/train.yaml \
    kpi_media_path=<dataset_root> \
    results_dir=<rca_results_dir> \
    top_k_per_label=50

Always pass `top_k_per_label`. This is the argument that switches the container

from the default "samples below threshold" filter into proper top-K-per-label

ranking. At min_recall=1.0 the threshold is by construction at-or-below every

NO_PASS score, so the below-threshold filter returns ONLY misclassified PASS rows

and zero NO_PASS rows — useless as an augmentation queue. With top_k_per_label

set to a positive integer (either in the spec or as a Hydra override), the

container computes signed weakness against the threshold for every row and

surfaces the K weakest per ground-truth label, which is the per-label ranked

output downstream steps consume.

Reads inference.csv, sweeps every unique siamese_score plus one value just below the minimum, keeps the candidates with NO_PASS-class recall ≥ min_recall (with 1e-12 tolerance), then picks the threshold with the best F1 (tie-break: precision, then threshold value). For every row, computes signed weakness from that threshold (positive = misclassified, negative = correct, magnitude = margin). Sorts by weakness descending and takes the top top_k_per_label per ground-truth label, then expands each weak row into one row per lighting condition using dataset.classify.input_map and dataset.classify.image_ext from the train YAML.

If no candidate threshold meets the recall target, the container exits non-zero and writes unreachable_kpi.txt into results_dir explaining which recall the model can actually achieve. In that case, stop the analysis after the docker call, write a one-section report explaining the model fundamentally cannot reach the KPI at any operating point, and recommend retraining or relabeling — skip the visual spot-check.

TAO 7.2 artifact contract:

Container-produced required artifacts:

You must produce the required seven-section RCA_Report.md after the container returns. On a reachable run, you must also produce rca_images/ during the visual spot check. The container does not produce either agent-owned artifact.

unreachable_kpi.txt is the failure artifact written when the recall target is unreachable. Its presence means: skip Step 5, write the abridged report, and recommend retraining. The TAO 7.2 vcn_aoi action does not emit metrics.json; that file is outside the 7.2 contract. Calculate report metrics from the full inference.csv and the chosen value in threshold.txt instead.

Print the container's stdout summary (chosen threshold, kept-row counts, per-label breakdown) to your own stdout so the script-check hook can verify the run produced output.

Step 5 — Visual spot check (small, fixed)

Skip this step if unreachable_kpi.txt exists. Otherwise use the Read tool to view the 5 weakest PASS samples and the 5 weakest NO_PASS samples from kpi_gaps.parquet (deduplicated to one row per sample, using the FIRST-lighting filepath), classify each as exactly one of mislabeled / edge case / data quality / systematic, and copy each viewed image (resized to 128×128 if PIL is available, otherwise just copy) into <results_dir>/rca_images/. This is the only image inspection required — do not view dozens of images, run failure mode clustering, or audit goldens (VCN has no golden images).

See references/visual-spot-check.md for the exact sample-selection sort, the per-lighting deduplication rule, the full definition of each verdict category, and the image-copy detail.

Reference invocation

Paste-and-edit the workspace, the four paths, and the two numeric knobs; this runs end-to-end. Capture stdout so the script-check hook sees row counts.

WORKSPACE=<absolute path>            # mounted identically inside the container
EXP_DIR=<experiment_result_dir>      # contains inference/inference.csv and train.yaml; must be inside $WORKSPACE
DATASET_ROOT=<dataset_root>          # image root for inference.csv input_path entries; must be inside $WORKSPACE
MIN_RECALL=1.0                       # zero-miss default; lower if KPI relaxes
TOP_K=50                             # per-label augmentation budget
OUT="$EXP_DIR/rca_results/$(date +%Y-%m-%d_%H%M%S)"
SPEC="$OUT/vcn_aoi_spec.yaml"
IMG="$(scripts/resolve_versions_key.py images.tao_toolkit.data_services)"
 
mkdir -p "$OUT"
 
# Write the gap-analysis spec for this run
cat > "$SPEC" <<EOF
min_recall: $MIN_RECALL
top_k_per_label: $TOP_K
EOF
 
docker run --gpus all --rm --ipc=host \
    -v "$WORKSPACE:$WORKSPACE" -w "$WORKSPACE" \
    "$IMG" gap_analysis vcn_aoi \
    -e "$SPEC" \
    inference_results_dir="$EXP_DIR/inference/latest/" \
    train_config="$EXP_DIR/train.yaml" \
    kpi_media_path="$DATASET_ROOT" \
    results_dir="$OUT"
 
# Container writes as root with --user dropped; chown back to host UID if needed.
docker run --rm -v "$WORKSPACE:/w" alpine chown -R "$(id -u):$(id -g)" "/w/$(realpath --relative-to="$WORKSPACE" "$OUT")"