Deterministic UI policy, discovery and quality gates for coding agents
OrchestrUI demonstrates solid tool design with clear naming, comprehensive parameter schemas, and structured output. All 7 tools follow verb_noun conventions (inspect_, list_, recommend_, get_, search_, audit_). Descriptions are substantive (100-500 chars), exceeding the 10-1024 baseline. Input schemas are well-typed with enums, length constraints, and nested objects. However, output schemas are not explicitly documented in the visible code, only inferred from implementation. Error handling is minimal; tools lack recovery guidance. The recommend_stack tool has a complex nested schema that could benefit from clearer parameter relationships documentation.
Score a proposed library plan against the nine OrchestrUI quality and licensing categories.
Return an official install command as inert text. This tool never executes shell or package-manager commands.
Return use, avoid, compatibility, legal and official-source guidance for one ecosystem.
Read bounded package.json, package-lock.json and components.json within the server working directory. Return a HostProfile with evidence; never execute project code.
List the seven supported UI ecosystems and optionally filter by role. Returns metadata only.
Compute a minimal UI composition from structured host/task profiles, with task text supported as a legacy inference path.
Output schemas not explicitly documented in tool definitions. LLMs cannot plan downstream calls without knowing return field names and types.
recommend_stack has deeply nested input schema (host_profile, task_profile objects with 10+ nested fields each). Parameter relationships (e.g., which fields are mutually exclusive) are not documented in descriptions.
No error recovery guidance. Tools lack descriptions of failure modes and what the LLM should do next (e.g., 'If library not found, call list_libraries() to see available options').
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 64 | 2025-06-18+ | v2 |
Search a selected library's public official registry when available, with a verified metadata-only fallback.
search_components accepts 'live' boolean but does not document fallback behavior or timeout handling in the description. LLMs cannot reason about retry strategy.
recommend_stack description mentions 'legacy inference path' but does not clarify when to use task text vs. structured profiles, or what the LLM should prefer.