Multi-agent BIM quality assurance orchestrator for AU 2026 with approval workflow, audit execution, and Revit/Forma integration
12 tools with mixed quality. Naming is verb-forward and clear (health, create_audit, list_runs, get_report). Descriptions exist but are terse (10-50 chars), below the 50-200 char LLM-optimized baseline. Input schemas are present for most tools but lack depth: many parameters have no type info visible (e.g., 'profile' in create_audit is typed 'object' with no nested schema). Output schemas are not documented anywhere in the source. Error handling is minimal, no recovery guidance, no categorization of retryable vs fatal errors. Security concern: create_audit accepts inline 'profile' object without validation hints. Composition is reasonable (single responsibility per tool), but audit_events returns SSE stream without documented structure. Overall: functional but below production baseline.
One ApprovalWatcher pass (K5 mirror) - execute pending approval writes
SSE progress stream for audit execution
Get job status (+ run summary when done)
Start an audit (409 while one runs, D7)
Get any run artifact (traversal-guarded)
Get run report in markdown format
Get verification report in markdown format
Probe the 4 capability axes (Revit, Forma, LOD, Spatial)
Output schemas not documented. Tools return markdown, JSON objects, and SSE streams with no schema definition visible. LLMs cannot plan downstream calls or extract structured data reliably.
Descriptions are terse (10-50 chars, well below 50-200 char baseline). 'SSE progress stream for audit execution' and 'List all audit runs' lack context on WHEN to call, WHAT to expect, and HOW to use results. LLMs cannot reliably select these tools.
Parameter schemas lack depth. 'profile' in create_audit is typed 'object' with no nested field definitions, constraints, or examples. LLMs cannot validate input or understand required vs optional fields.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 55 | 2026-07-28+ | v2 |
Mark an approval proposal as ignored
List all approval proposals with their status and fixes
List all audit runs
Restore an ignored approval proposal back to pending
No error handling guidance. Tools have no documented error responses, recovery paths, or categorization (retryable vs fatal). A 409 on create_audit is mentioned in description but not in error schema.
audit_events returns SSE stream with undocumented event structure. No schema for event payloads, field names, or timing. LLMs cannot parse or act on progress updates.