carla-simulator/carla-agentic-tools/bounding-boxes

Bounding boxes

Paths. scripts/… and references/… below are relative to the

directory holding this SKILL.md. Your working directory is the user's

project, not that directory, so prefix them with its absolute path or the

command is not found.

Three things: read actors' 3D boxes, draw them in the world, or project them into a camera image as 2D boxes with a JSON sidecar — the standard CARLA dataset-annotation pipeline (client_bounding_boxes.py).

Instructions

Progress:
- [ ] Step 1: Check prerequisites (bash scripts/check_env.sh), clear FAILs
- [ ] Step 2: For `project`, have a camera on the ego (create-sensor) + actors around
- [ ] Step 3: list / draw / project

Commands need CARLA_HOST/CARLA_PORT from scripts/env.sh.

Step 1: Check prerequisites

bash scripts/check_env.sh

Step 3: Boxes

source scripts/env.sh
 
# 3D box geometry of every vehicle
python3 scripts/bounding_boxes.py list --filter vehicle.*
 
# draw 3D boxes around vehicles — they track moving cars, stay 30 s (default)
python3 scripts/bounding_boxes.py draw --filter vehicle.*
python3 scripts/bounding_boxes.py draw --filter vehicle.* --seconds 60
 
# project vehicle boxes into a camera frame -> annotated PNG + JSON
python3 scripts/bounding_boxes.py project --camera <camera_id> --filter vehicle.* --out boxes.png

How the projection works (brief)

Capture one frame → build the camera intrinsics K from its fov + resolution → for each actor, take its 3D box's 8 world vertices (get_world_vertices), transform to camera space with the camera's inverse matrix, keep those in front, apply K → 2D points → the min/max is the 2D box. Off-frame and too-far (--max-dist) actors are skipped. Full math in the reference.

Examples

Example 1: annotate the ego camera

User says: "put boxes on the cars in the camera view"

Spawn a camera on the ego (create-sensor), then project --camera <id> --filter vehicle.* → boxes.png with green 2D boxes + boxes.json.

Example 2: how big is that actor

User says: "what are the dimensions of vehicle 137?"

list --filter vehicle.* (find 137) → its LxWxH from the extent.

Example 3: visualise in 3D

User says: "outline all the vehicles in the world"

draw --filter vehicle.* — green 3D boxes on a rendered server.

Troubleshooting

Problem: `project` finds no boxes Cause: no actors in front of / within --max-dist of the camera, or wrong filter. Solution: raise --max-dist, widen --filter, ensure vehicles are in view.

Problem: boxes misaligned Cause: the frame and the actor poses drifted (fast motion, async). Solution: capture in synchronous mode (set-world-settings) so the frame and poses are from the same tick.

Problem: `draw` shows nothing Cause: headless -nullrhi server. Solution: use a rendered server (see debug-draw).

Note: boxes always track moving actors draw re-stamps each frame for --seconds (default 30), so boxes ride moving cars automatically — the command blocks for that duration.

Outputs

• list: box dimensions per actor.
• draw: 3D box overlay (rendered view).
• project: <out>.png (annotated) + <out>.json (2D boxes: id, type, xyxy).

Detail (intrinsics, world→camera transform, level boxes, lidar-to-camera) in references/bounding-boxes.md.