Loading ToolBench
MCP server exposing analyze_task_complexity: routes a coding task to Haiku or Sonnet using the same LOCALIZED/DIAGNOSTIC classifier as harness/route_task.py
Single tool 'analyze_task_complexity' has a well-written description (194 chars, within baseline 34-392 range) and clear purpose. Input schema is present with Zod validation and parameter descriptions. However, output schema is NOT formally documented, the tool returns a JavaScript object with fields like 'classification', 'confidence', 'reasoning', 'parse_error', 'recommended_model', etc., but no JSON Schema or TypeScript interface is exposed to the MCP client. This forces LLMs to infer output structure. Error handling is present but generic ('analyze_task_complexity failed: {err.message}'). No tool annotations (readOnlyHint, destructiveHint, idempotentHint). The tool is READ_ONLY and idempotent, but these properties are not declared in the schema.
Classifies a coding task prompt as LOCALIZED (the fix location is already implied - safe to route to a cheap model) or DIAGNOSTIC (needs cross-file investigation - route to a stronger model). Runs the same classifier as this project's harness/route_task.py, including its conservative confidence threshold: any uncertainty falls back to the stronger model rather than risking a silently-wrong fix on the cheap one. Call this BEFORE running an expensive multi-step coding task to get a model recommendation.
Output schema not formally documented. Tool returns a complex object with 7+ fields (classification, confidence, reasoning, parse_error, recommended_model, routing_note, classifier_cost_usd, raw_result_text) but no JSON Schema or TypeScript interface is exposed to the MCP client. LLMs must infer structure from unstructured text responses.
Tool annotations missing. 'analyze_task_complexity' is READ_ONLY and idempotent (safe to retry), but these properties are not declared via readOnlyHint or idempotentHint in the schema. Agents cannot infer safety properties without explicit hints.
Error handling is generic. When the classifier CLI fails or JSON parsing fails, the tool returns a bare error message ('analyze_task_complexity failed: {err.message}') with no recovery guidance. Per pattern:recovery-guide, errors should suggest next steps (e.g., 'Classifier CLI not found. Ensure claude CLI is installed and in PATH.').
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 54 | 2025-06-18+ | v2 |
Parameter 'file_tree' is optional but its utility is not clearly explained. Description says 'Improves classification accuracy but is not required', but by how much? When should an agent include it? Guidance is vague.
No pagination or result limits documented. While this tool returns a single analysis object (not a list), the 'raw_result_text' field could be large. No guidance on maximum response size or truncation behavior.