Loading ToolBench
MCP server that exposes Pyright language server functionality
Four tools with clear verb-noun naming (check_types, get_hover, get_completions, get_definition) and reasonable descriptions (80-150 chars). However, critical gaps: input schemas lack proper type declarations for several parameters (python_path, type_checker_mode are strings but lack enum constraints); position parameter lacks range/bounds documentation; output schemas are not formally documented; error handling is minimal (no recovery guidance); no tool annotations (readOnlyHint present but not declared). The get_hover implementation is incomplete (truncated in source). Parameters are described but lack format constraints and validation rules. Baseline: avg tool description ~110 chars (acceptable), but schemas incomplete and error guidance absent.
Check types in a Python file or string of code. Performs type checking using Pyright and returns diagnostics.
Get code completions at a specific position. Returns available completions at the given cursor position.
Get definition location at a specific position. Returns the location of the definition for the symbol at the given position.
Get hover information at a specific position. Returns type information and documentation for the symbol at the given position.
Input schemas lack enum constraints for type_checker_mode parameter. Accepts 'basic', 'strict', 'overload' but no enum declaration, LLMs may hallucinate invalid values like 'pedantic' or 'lenient'.
Output schemas not formally documented. Tools return dicts with fields like 'diagnostics', 'range', 'contents', 'kind' but no JSON Schema definition visible. LLMs cannot reliably extract or chain results.
Error handling lacks recovery guidance. RuntimeError for missing pyright, ValueError for missing params, but no actionable next steps. E.g., 'Pyright not found. Install with: pip install pyright' is good, but other errors (parse failures, timeouts) return bare dicts with 'error' key.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | 2026-07-28+ | v2 |
Position parameter (get_hover, get_completions, get_definition) lacks bounds documentation. No min/max specified; LLMs may pass negative or out-of-range positions. Should document: '0-indexed character position, must be <= len(code)'.
Tool annotations missing. All four tools are read-only (no state modification) but readOnlyHint is not declared in schema. Agents cannot infer safety/idempotency without explicit annotations.