wu-yc/labclaw/gaussian-splatting-scene-description

Gaussian Splatting Scene Description

Overview

gaussian_splatting_scene_description bridges 3D lab reconstruction and natural language understanding. Given a small set of lab photos or short video clips, it builds a 3D Gaussian Splatting (3DGS) scene representation and then generates a structured natural language description of the spatial layout — instrument positions, sample locations, bench topology, and relational predicates (e.g., "pipette is left of tube rack", "centrifuge is behind the operator"). The output is designed for downstream consumption by VLMs, spatial reasoning models, or LabOS skills (protocol_video_matching, detect_common_wetlab_errors, realtime_protocol_guidance_prompts) that need a persistent, queryable representation of the lab environment for context-aware guidance, error detection, or AR overlay anchoring.

When to Use This Skill

Use this skill when any of the following conditions are present:

• Spatial context for protocol guidance: A protocol step references "the tube on your left" or "the centrifuge behind you"; the agent needs a 3D-aware scene description to resolve spatial references and generate accurate realtime_protocol_guidance_prompts.
• Lab layout documentation: A new lab setup or bench configuration must be documented in natural language for onboarding, SOP writing, or remote collaboration — "The pipette is on the right side of the bench; the tube rack is centered; the centrifuge is 2 m behind."
• VLM / spatial model pre-training or fine-tuning: A VLM or spatial intelligence model requires structured scene descriptions as training data; the skill produces consistent, schema-aligned text from real lab imagery.
• AR overlay anchoring: AR overlays (e.g., step indicators, deviation highlights) need to be anchored to 3D positions; the scene description provides object labels and approximate coordinates for overlay placement.
• Multi-session consistency: The same lab is imaged at different times; 3DGS + description enables comparison ("centrifuge was moved from left to right since last scan").
• Error detection context: detect_common_wetlab_errors or protocol_video_matching benefits from knowing the canonical layout — e.g., "tube A1 is in position (x, y) of the rack" — to disambiguate observations.
• Robotic or automation planning: A lab robot (Opentrons, Hamilton) or future automation system needs a natural language map of the workspace for path planning or object localization.
• Virtual lab tours or training: Generate descriptive text for a 3D lab model used in VR training, virtual tours, or remote supervision.

Core Capabilities

1. 3D Gaussian Splatting Reconstruction

Builds a 3D scene representation from sparse input:

• Input formats: 5–50 lab photos from different angles, or a short video clip (10–60 s) with sufficient camera motion; supports JPEG, PNG, MP4, MOV
• Reconstruction pipeline: COLMAP or similar SfM for camera pose estimation → 3DGS training (e.g., gsplat, nerfstudio 3DGS backend, or custom implementation) → optimized Gaussian splat cloud
• Output: 3DGS model file (.ply, .splat, or framework-specific format); camera poses; sparse point cloud (optional)
• Requirements: Overlapping views, sufficient parallax; textured surfaces; avoid pure reflective or transparent objects (common lab glassware may need special handling)
• Compute: GPU recommended (NVIDIA with CUDA); typical runtime 2–10 min for 20–30 images on consumer GPU
• Lab-specific tuning: Bench surfaces, instrument housings, and sample containers typically reconstruct well; fine tubing and labels may be noisy — description step filters and abstracts

2. Scene Understanding & Object Detection

Extracts semantic and spatial information from the 3DGS scene:

• Object detection: Runs 2D object detection (YOLO, DETR, or VLM-based) on input images; back-projects detections into 3D using camera poses and depth (from 3DGS or depth estimation)
• Lab object vocabulary: Predefined classes — pipette, tube_rack, eppendorf_tube, centrifuge, microscope, incubator, plate, reagent_bottle, balance, vortex, ice_bucket, bench, sink, hood, robot_arm — extensible via custom labels
• 3D bounding boxes: Each detected object gets an approximate 3D bounding box (center xyz, extent); merged across views for consistency
• Spatial clustering: Groups nearby objects (e.g., tubes in a rack) into logical units; assigns IDs (A1, B2, etc.) when grid layout is detected
• Relational extraction: Computes pairwise relations — left-of, right-of, in-front-of, behind, above, below, inside (e.g., tube in rack) — from 3D positions and viewing conventions

3. Natural Language Description Generation

Produces structured text from the 3D scene graph:

• Description schema:
• Layout summary: One-paragraph overview of the scene (bench type, dominant instruments, general arrangement)
• Object list: Each detected object with label, approximate position (e.g., "center-left", "right third"), and optional 3D coordinates
• Relational predicates: "The pipette is to the left of the tube rack." / "The centrifuge is 1.5 m behind the bench."
• Spatial keywords: Curated vocabulary for downstream models — bench_center, left_zone, right_zone, behind_operator, above_bench, in_hood, on_ice
• Sample/container map: For tube racks or plates, "Tube A1 at (x, y); Tube B2 at (x, y); …" or "96-well plate centered; columns 1–6 visible"
• Granularity levels: brief (3–5 sentences), standard (full schema), verbose (includes confidence scores, alternative interpretations)
• Coordinate system: Defines origin (e.g., bench center, operator position); reports positions in meters or normalized 0–1; includes coordinate frame description for downstream use
• Temporal consistency: When input is video, optionally tracks object stability across frames; notes "object X moved between frames 10 and 50" if significant displacement detected

4. Output Formats

Emits description in multiple formats for different consumers:

• Plain text: Flowing paragraph + bullet list of objects and relations
• Structured JSON: Schema-validated JSON with objects[], relations[], layout_summary, coordinate_frame, spatial_keywords[]
• RDF triples (optional): Subject-predicate-object triples for knowledge graph integration
• VLM prompt prefix: Formatted as a context block to prepend to VLM queries — "Scene: The bench has a pipette on the left, a tube rack in the center, and a centrifuge behind. Tube A1 is at position (0.2, 0.3). Question: …"
• Markdown: For documentation or ELN embedding; includes optional ASCII layout sketch

5. Lab-Specific Enhancements

Optimizes for wet-lab and instrumentation contexts:

• Instrument recognition: Fine-tuned or prompt-based recognition for common lab equipment (Eppendorf pipettes, Thermo centrifuges, Zeiss microscopes, etc.); adds model/vendor to description when identifiable
• Sample grid inference: Detects 96-well plates, tube racks (8×12, 4×6); infers well IDs from position
• Hazard zone tagging: Marks regions (hood, centrifuge, hot plate) with hazard keywords for safety-aware downstream models
• Lighting and occlusion: Notes "left side of bench in shadow" or "tube rack partially occluded by bottle" when reconstruction quality varies
• Scale calibration: Uses known object size (e.g., standard 1.5 mL Eppendorf) to estimate scene scale when no explicit scale is provided

6. Integration with LabOS Pipeline

Feeds into downstream skills:

• `protocol_video_matching`: Scene description provides "where is the tube for step 5?" context; protocol step "add to tube A1" is resolved using tube_rack → A1 position
• `realtime_protocol_guidance_prompts`: "The reagent is on your left" — generated from scene description + operator head pose
• `detect_common_wetlab_errors`: Expected instrument positions help disambiguate "pipette in wrong hand" vs. "pipette in correct position"
• AR overlay systems: Object 3D positions and labels for spatial anchoring of UI elements
• `extract_experiment_data_from_video`: ROI suggestions from scene description ("tube_rack zone: x1,y1,x2,y2")

Usage Examples

Example 1 — Bench Layout from 15 Photos

Input:

INPUT:
  images:       ["bench_01.jpg", "bench_02.jpg", ..., "bench_15.jpg"]  # 15 photos around bench
  output_format: "json"
  granularity:  "standard"
 
→ COLMAP: 15 poses, 8k points
→ 3DGS training: 3 min on RTX 3080
→ Object detection: pipette, tube_rack, centrifuge, 3 reagent bottles, ice bucket
→ Back-projection: 3D positions for each object
→ Relation extraction: pipette left-of tube_rack; centrifuge behind bench; ice_bucket right-of tube_rack

Output (JSON excerpt):

{
  "layout_summary": "A standard lab bench with a tube rack centered, pipette on the left, and ice bucket on the right. A centrifuge is visible 1.2 m behind the bench. Three reagent bottles are arranged along the back edge.",
  "coordinate_frame": {
    "origin": "bench_center",
    "x_axis": "left_to_right",
    "y_axis": "front_to_back",
    "z_axis": "up",
    "units": "meters"
  },
  "objects": [
    {"id": "pipette_1", "label": "pipette", "position": [-0.25, 0.1, 0.05], "zone": "left_zone", "notes": "single-channel, likely P200"},
    {"id": "tube_rack_1", "label": "tube_rack", "position": [0.0, 0.15, 0.02], "zone": "bench_center", "grid": "8x12", "notes": "standard 96-tube rack"},
    {"id": "centrifuge_1", "label": "centrifuge", "position": [0.0, 1.2, 0.5], "zone": "behind_operator", "notes": "bench-top model"}
  ],
  "relations": [
    {"subject": "pipette_1", "predicate": "left_of", "object": "tube_rack_1"},
    {"subject": "centrifuge_1", "predicate": "behind", "object": "bench"},
    {"subject": "ice_bucket_1", "predicate": "right_of", "object": "tube_rack_1"}
  ],
  "spatial_keywords": ["bench_center", "left_zone", "right_zone", "behind_operator", "on_ice"]
}

Example 2 — VLM Prompt Prefix for Protocol Step

Input:

INPUT:
  scene_json:   "lab_bench_scene_2026-03-06.json"  # from Example 1
  output_format: "vlm_prompt_prefix"
  max_tokens:   150
 
→ Format as context block for VLM

Output:

Scene: Lab bench. Tube rack (8×12) centered. Pipette on left. Ice bucket on right. Centrifuge behind bench. Reagent bottles along back. Tube A1 at (0.0, 0.15). Coordinate origin: bench center. Units: m.

Example 3 — Short Video Clip → 3DGS + Description

Input:

INPUT:
  video:        "bench_walkthrough_30s.mp4"  # 30 s, operator walks around bench
  sample_frames: 20  # extract 20 frames for reconstruction
  output_format: "markdown"
 
→ FFmpeg: extract 20 frames at uniform intervals
→ COLMAP + 3DGS: reconstruct scene
→ Object detection on keyframes; merge
→ Generate Markdown description with ASCII layout sketch

Output (Markdown excerpt):

## Lab Bench Layout (from 30 s video)
 
**Summary:** The bench has a tube rack in the center, pipette on the left, and a vortex mixer on the right. A microscope is visible at the far end.
 
**Object Map:**
| Object    | Position    | Zone        |
|-----------|-------------|-------------|
| Pipette   | (-0.3, 0.1) | left        |
| Tube rack | (0, 0.15)   | center      |
| Vortex    | (0.3, 0.1)  | right       |
| Microscope| (0.4, 0.8)  | far_right   |
 
**Layout sketch (top-down):**

[Microscope]

[Pipette] -- [Tube Rack] -- [Vortex]

[Bench surface]

Integration Notes