reality-opened/openreality/openreality

Open Reality OS

You have MCP tools (server name openreality) for a user's Open Reality account — persisted 3D scenes reconstructed from phone captures, imported splats, and robot recordings.

Workflow

1. Sync first: workspace_list_scenes — the account is the sync layer; scans made on

the user's phone or the web appear here. new_since_last_sync tells you what's new.

1. Context second: scene_card <scan_id> — small, honest summary (source, metric

state, objects, artifacts). @-mention openreality://scene/<scan_id> works too.

1. Then act:

• Deterministic questions (distances, angles, room structure, paths, exports) →

scene_* / export_* tools. These are pure server routes — no LLM cost.

• Narrated / visual questions ("describe this room", "key features") →

agent_chat / agent_annotate. These run the SERVER-side scene agent, which spends a bounded LLM budget from the account. Prefer agent_replay (always $0) when a recorded run already answers the question (agent_runs lists them).

1. Uploads: workspace_upload_video (≤1 GB) / workspace_upload_recording (robot

.db) / workspace_upload_splat (.ply/.spz) → then workspace_job_wait on the job_id.

1. Artifacts go to disk, never into context: artifact_fetch / artifact_fetch_splat

return local file paths. Exports: export_manifest (dry-run) → export_prepare → artifact_fetch of the returned key. Formats: openreality, groot_lerobot_v2, isaac_usd, mujoco_mjcf. For MuJoCo, export_mujoco does all of it in one call and returns the unpacked scene.xml path. Both simulator formats need a metric anchor (scene_anchor first); MuJoCo collision is a floor plane + detected-object boxes only.

Honesty doctrine (non-negotiable — mirrors the product's own guardrails)

• Units: a length is metres ONLY when the response says units: "m" (a metric anchor

was applied — scale_source names it). units: "relative" means relative SLAM units: never call them metres, never convert. To unlock metres: scene_anchor with two picked points and their real distance.

• Closed world: the objects from scene_list_objects are the only objects that exist.

Don't invent, merge, or rename them beyond their labels.

• Provenance rides along: keyframes are photos; synthetic views are RENDERS of the

splat (say so); generated assets (SAM-3D completions, variants, floor patches) are generated — their meta.json says generated: true. Nav plans carry "not certified navigation" — repeat it when presenting a path.

• Degraded is a state, not a failure: robot-recording scenes persist degraded (no

detections, preview cloud) with a QC report under derived/demo/recordings/ — report the QC verdict rather than hiding it.

• The Isaac export is metric-gated: without an anchor it is refused. That refusal is

correct behavior — offer the anchor flow, don't fight the gate.

• Surface server refusals as-is: 409 agent_run_active (one run per scene — attach or

wait), 422 unreachable_goal (includes the nearest reachable point), 409 export_job_active.

Troubleshooting

• No Open Reality credentials → run openreality-mcp login --token <token> (token from

the deployment's /os page URL hash after dashboard sign-in) or set OPENREALITY_TOKEN.

• Tokens roll automatically at half-life; a stale (>12 h idle) credential needs a fresh

login.

• To develop without the real backend: openreality-mcp simulator, then point the server

at it with OPENREALITY_URL=http://127.0.0.1:8973 OPENREALITY_TOKEN=sim-token.