nvidia-tao/tao-skill-bank/tao-analyze-gaps-od-map

TAO Analyze Gaps OD mAP

Use this skill to run TAO Data Services object-detection gap analysis. The skill compares ground-truth and inference annotations, computes per-image per-class TP/FP/FN/AP50 metrics, and identifies weak images where any class metric falls below its threshold. It does not run inference; upstream steps must produce the inference annotations first.

The container entrypoint is:

gap_analysis object_detection -e /absolute/path/to/object_detection.yaml

Inputs

Required spec fields:

Common optional fields:

Do not hand-write the spec. Copy the template and fill in the nulls — every tuning value it already carries is the one this stage wants — then validate:

cp skills/data/tao-analyze-gaps-od-map/assets/default_object_detection.yaml "$SPEC"
# fill ground_truth_ann_path, inference_ann_path, images_dir, results_dir, kpi, input_format
python3 skills/data/tao-analyze-gaps-od-map/scripts/verify_object_detection_spec.py --spec "$SPEC"

verify rejects the spellings that fail — uppercase input_format, relative or missing paths, a weak_thresholds entry that is a bare number rather than a mapping — and reports every gated class plus the default_* fallbacks, so the selection criteria behind a weak set are recoverable from the run's output. It warns when a fallback is above zero, since that gates classes you did not list.

Quick Start

Run from the tao-skill-bank repo root.

Write the spec into the results directory. The run emits four artifacts and does not retain the spec, so a completed gap analysis otherwise cannot tell you which thresholds produced its weak set — and that weak set sizes the mining budget downstream. Keeping them together makes the selection criteria recoverable from the run alone.

RESULTS_DIR=/absolute/path/for/this/run          # results_dir in the spec
SPEC="$RESULTS_DIR/object_detection.yaml"        # spec lives beside its outputs
RUN_ROOT=/absolute/path/that/contains/annotations/images/and/results
GPU_COUNT=1
 
DS_IMAGE="$(scripts/resolve_versions_key.py images.tao_toolkit.data_services)"
 
RUNTIME_HOME="$RUN_ROOT/.tao-runtime/home"
mkdir -p "$RUNTIME_HOME/.cache"
 
docker run --rm --gpus "$GPU_COUNT" --ipc=host --network=host \
  --user "$(id -u):$(id -g)" \
  -e USER="$(id -un)" -e LOGNAME="$(id -un)" \
  -e HOME="$RUNTIME_HOME" -e XDG_CACHE_HOME="$RUNTIME_HOME/.cache" \
  -v "$RUN_ROOT:$RUN_ROOT" \
  -w "$RUN_ROOT" \
  "$DS_IMAGE" \
  gap_analysis object_detection -e "$SPEC"

Keep --user and the identity env block above. TAO DS images call getpass.getuser() at startup; without USER/LOGNAME they fail with KeyError: 'getpwuid(): uid not found: <uid>' because the UID has no /etc/passwd entry. Do not work around it by running the container as root; that leaves root-owned outputs. This follows the launch contract in tao-run-on-docker.

Preflight

1. Verify Docker access:

docker info > /dev/null

1. Resolve and pull the data-services image if needed:

DS_IMAGE="$(scripts/resolve_versions_key.py images.tao_toolkit.data_services)"
docker image inspect "$DS_IMAGE" > /dev/null || docker pull "$DS_IMAGE"

1. Confirm RUN_ROOT contains the spec, both annotation sources, and the image directory. Mount RUN_ROOT to the same absolute path inside Docker.

Outputs

All four artifacts are always written, even when no gaps are found.

Troubleshooting

`The subtask object_detection requires -e/--experiment_spec_file`: rerun with gap_analysis object_detection -e "$SPEC".

Input path not found inside Docker: use a RUN_ROOT mount where host and container paths are identical.

`input_format` error: set input_format: kitti or input_format: coco explicitly — it is never inferred from the path.

`weak_images.parquet` is empty: all class metrics are above their thresholds. Lower default_recall_threshold / default_ap50_threshold or add per-class entries to weak_thresholds.

Output directory not writable after Docker exits: the container ran without --user, so it wrote as root. Relaunch with the --user and identity env block shown above.