Multi-model AI consensus system — works with any IDE. Exposes 35 tools for cross-examination, A/B testing, research, and capability management across cardiologist and neurologist LLM providers.
38 tools with inconsistent quality. Most have basic descriptions (50-150 chars) and input schemas, but lack depth, error guidance, and output documentation. Naming is verb-forward but often generic (e.g., 'ask_local_tool', 'ask_remote_tool'). No tool descriptions explain WHEN to use them vs. alternatives, what they return, or how to handle failures. Parameters lack validation constraints (enums, ranges, formats). No evidence of output schemas or pagination support. Security-sensitive tools (ab_start, cap_ab_start, event_publish) lack permission gates or audit trail documentation. Composition is poor: multiple tools do similar things (consult vs. cross_examine, ask_local_tool vs. ask_remote_tool, ab_* vs. cap_ab_*) without clear differentiation.
Conclude an A/B test with verdict
Record A/B test measurement
Propose an A/B test experiment
Start A/B test grace period
Quick 3-surgeon fix validation
Direct query to neurologist
Direct query to cardiologist
Duplicate/overlapping tool functionality without clear differentiation. ask_local_tool vs ask_remote_tool, ab_* vs cap_ab_*, consult vs cross_examine all perform similar operations. LLMs cannot disambiguate and waste reasoning cycles.
No output schemas documented. Tools return unstructured responses (inferred from code: dicts with 'stream_id', 'events', 'removed' fields). LLMs cannot plan downstream calls or extract required fields reliably.
Descriptions lack WHEN/WHY guidance. 'Scan for complexity risk vectors' (sentinel_run) does not explain when to call it vs other analysis tools, what 'complexity risk' means, or what the output contains. Descriptions average 40-50 chars; production baseline is 194 chars.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 53 | 2026-07-28+ | v2 |
Multi-surgeon A/B test design
Conclude an A/B test with verdict
Measure an active A/B test
List A/B tests in the queue
Start an A/B test
Veto an A/B test
Multi-surgeon evidence reverification
List available chain presets
Execute a chain preset
Show mode suggestions for trigger
4-phase deep audit pipeline
Cross-check evidence with LLM analysis
Research budget and cost tracking
System health and capability overview
Cardiologist cross-examination review
Confidence-weighted consensus on a claim
Quick consult with both surgeons
Full cross-examination protocol with iterative review support
Poll for events on subscription stream
Publish event to IDE event bus
Subscribe to IDE event bus patterns
Unsubscribe from event stream
Infrastructure health gate verification
Surgeon capability self-report
Corrigibility skeptic challenge
Neurologist system health pulse check
Health check all 3 surgeons
Self-directed research on a topic
Scan for complexity risk vectors
Show upgrade event log
Probe ecosystem and report detected phase
No error handling or recovery guidance. Tools lack descriptions of failure modes, retryability, or what the LLM should do if a call fails. No validation constraints (enums, ranges) on parameters like 'depth', 'mode', 'verdict'.
WRITE tools lack permission gates and audit trail documentation. ab_start, cap_ab_start, event_publish, cap_chain_run modify state but have no documented permission checks, scope declarations, or audit logging. Violates security baseline.