Loading ToolBench
Second-opinion advisor on an external model chosen by server config. Provides expert analysis and professional judgment on technical issues, workflows, and systems via MCP.
Single tool 'consult' has a well-written description (194 chars, within baseline 34-392 range) that clearly states WHAT it does, WHEN to use it, and prerequisites. Input schema is present with three parameters (question, context, workhorse), all with descriptions. However, parameter types lack explicit JSON Schema type declarations in the visible code, they are inferred as strings from context. Output schema is not documented; the tool returns free-form text from an external LLM advisor with no structured response definition. Error handling is present (RuntimeError for missing ADVISOR_MODEL) but lacks recovery guidance. No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite the tool being read-only and idempotent.
Second-opinion advisor on an external model chosen by server config (ADVISOR_PROVIDER / ADVISOR_MODEL). The preferred consult route for architectural or hard-to-reverse decisions, twice-failed fixes, and tradeoffs unresolvable from the task. Send one focused question; put options, constraints, and what was already tried in context. The advisor is stateless and sees only what you send. Never consult it for routine work you can complete yourself.
Output schema not documented. Tool returns free-form text from external LLM; no structured response definition provided. LLMs cannot plan downstream actions or extract specific fields.
Parameter types not explicitly declared in JSON Schema format. Code infers strings from context, but schema lacks formal 'type' declarations. Parameters should declare type, minLength, maxLength, and enum constraints.
Error messages lack recovery guidance. RuntimeError for missing ADVISOR_MODEL is clear, but other failure modes (API timeout, invalid provider, auth failure) do not guide the LLM on next steps.
Tool annotations missing. 'consult' is read-only and idempotent but does not declare readOnlyHint or idempotentHint. Agents cannot infer safety properties without explicit annotations.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 63 | 2026-07-28+ | v2 |
No pagination or result limits documented. If external advisor returns very long responses, no mechanism to cap or stream output. Large responses risk context window exhaustion.