Open Reality OS as MCP tools: drive scan upload, reconstruction, scene query, scene agents, and robot-training exports from Claude Code, Claude desktop, Codex, or Cursor.
OpenReality MCP presents a moderately well-structured tool set for 3D scene manipulation and robotics workflows. Tool naming follows verb_noun conventions consistently (workspace_list_scenes, scene_locate_object_3d, agent_run). Input schemas are complete and well-typed across all 17 tools, with proper use of enums (e.g., export_dataset's format enum) and array constraints (scene_search_objects, scene_infer_spatial_relations). However, descriptions are uniformly terse (averaging ~45 chars, well below the 194-char production baseline), lacking context about WHEN to use each tool, prerequisites, or recovery guidance. Error handling is present (ApiError wrapper in toolkit.ts shows structured error surfacing), but descriptions do not guide LLM recovery. Parameter descriptions are minimal, many are one-line restatements of the name (e.g., 'The scene ID' for scene_id). No explicit output schemas are documented, forcing LLMs to infer response structure. The server demonstrates solid engineering (Zod validation, proper HTTP abstraction, guard pattern for error isolation), but falls short of production-grade agent optimization.
Get the status of a running agent job
Run the scene agent with a natural language instruction
Fetch an artifact (image, file, etc.) from the server
Export the scene as a robot-training dataset
Export the scene as an Isaac Sim/Lab USD file
Export the scene as a 3DGS splat point cloud
Get scene metadata and preview
Infer spatial relationships between objects
Descriptions uniformly under 50 characters, far below 194-char production baseline. Most are terse one-liners like 'Get scene metadata and preview' without context on WHEN to use, prerequisites, or what distinguishes this tool from similar ones (e.g., scene_card vs scene_list_objects). LLMs cannot distinguish tools or infer proper sequencing without fuller docstrings.
Parameter descriptions lack expected format, range, or constraint guidance. E.g., scene_id parameters are described only as 'The scene ID' with no hint that it must be obtained from workspace_list_scenes or scene_card first. timeout_s and poll_s in workspace_job_wait lack min/max bounds (e.g., 'Timeout in seconds (default 900)' should say 'Timeout in seconds: 1-3600 (default 900)'). This violates the pattern:constrained-input requirement.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 55 | 2026-07-28+ | v2 |
Get detailed information about a specific detection
List all detected objects in a scene
Get 3D location of an object in the scene
Measure distance between two 3D points in the scene
Plan a navigation path in the scene
Search for objects in a scene by query
Poll and wait for a workspace job to complete
List all scenes in the workspace
Upload a scan archive to create a new scene
No documented output schemas. Tool result shapes are inferred from code (toolkit.ts imageResult, jsonResult, textResult wrappers), but LLMs have no spec of what fields to expect. E.g., scene_card presumably returns metadata fields, but which ones? Does it return object_count, preview_url, mesh_available? LLMs must guess, leading to incorrect downstream tool chains and wasted context.
Error handling guidance is missing from descriptions. toolkit.ts shows structured ApiError handling (status, body, message), but descriptions do not guide LLMs on recovery. E.g., if scene_search_objects returns 'Scene not found', does the LLM retry with workspace_list_scenes, or is it fatal? No guidance provided.
Destructive operations (workspace_upload_scan, agent_run, export_dataset, export_isaac_scene, export_splat) lack confirmation or dry-run guidance. Descriptions do not warn LLMs that these calls have side effects (create new scenes, run agents, consume compute resources, trigger exports). No mention of idempotency or whether retrying is safe.
scene_search_objects and scene_infer_spatial_relations accept queries as arrays, but description does not explain query format. Are queries natural language ('red cube'), object class names ('cube', 'cylinder'), or semantic labels? This ambiguity forces LLMs to guess and retry on failure.
Numerical arrays (point_a, point_b in scene_measure_distance; start, goal in scene_plan_path) lack explicit documentation that they are [x, y, z] coordinates in SLAM units, not meters. No mention of coordinate system, scale assumptions, or units. LLMs cannot reason about scale transformations without this context.
agent_run max_iterations parameter lacks guidance on semantics. What does 'iteration' mean? Is it tool call, reasoning step, or loop of agent planning? Unbounded numbers can cause runaway execution. Description should specify typical range (e.g., '1-50') and costs associated with high values.