Loading ToolBench
Professional Python verification CLI tool combining linting, type checking, and smart testing
Server has 5 tools with complete input schemas and descriptions. Naming follows verb_noun convention (verify, graph, affected, regen_graph, list_tasks). Descriptions are present (avg ~120 chars) but lack actionable context for LLM selection. Parameters are typed but many lack descriptions (e.g., 'tasks' array items have no guidance on valid task names). Output schemas are not documented, LLMs cannot plan downstream calls. Error handling and recovery guidance are absent. No tool annotations (readOnlyHint/destructiveHint) despite clear risk levels. Composition is reasonable: tools are single-purpose and chainable (e.g., affected → verify). Security: regen_graph is marked WRITE but no permission checks or audit trail visible.
List tests affected by code changes (via py-smart-test). Returns JSON with affected test modules.
Build and return the dependency graph for the project. Returns JSON with all modules, their imports, and any circular dependencies.
List all available verification tasks with their metadata. Returns JSON array of task objects with name, category, description, etc.
Regenerate the py-smart-test dependency graph. Returns JSON with success/failure status.
Run Python verification checks (linting, type checking, testing). Returns structured JSON with all step results, issues found, and timing.
Output schemas not documented. LLMs cannot infer what fields to expect from verify, graph, affected, or list_tasks responses, blocking downstream tool chaining and data extraction.
Parameter 'tasks' in verify tool lacks description of valid task names (quality, tests, ruff, mypy, etc.). LLMs cannot determine which tasks to request without trial-and-error.
No tool annotations (readOnlyHint/destructiveHint/idempotentHint) despite clear risk levels. regen_graph is marked WRITE but LLM has no signal to treat it as destructive or require confirmation.
Error handling and recovery guidance absent. Tools return JSON but no documentation of error fields, failure modes, or what LLM should do on failure (retry, ask user, abort).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 55 | 2026-07-28+ | v2 |
regen_graph (WRITE operation) has no permission checks, audit trail, or confirmation step visible. Agents could trigger graph regeneration without authorization or logging.