Loading ToolBench
Reviewable completion-evidence workflow, bounded scoring, guided hardening, and learning-plan system for AI-built apps. Stdlib-only Python CLI plus a stdio MCP server with 11 tools; no network, no telemetry, secret values never printed.
CheckYourself has 11 tools with complete input schemas and reasonable descriptions, but several critical gaps limit production readiness. All tools have descriptions (10-150 chars avg), and all expose input parameters with type definitions. However, NO tools declare output schemas, and error handling guidance is minimal. The server implements a specialized audit workflow (scan→coverage→score→backlog→batch) rather than generic CRUD operations, which trades flexibility for domain focus. Tools like 'scan', 'coverage', 'score', and 'dashboard_data' are well-named and verb-based, but tools like 'capabilities' and 'receipt' lack clarity on when/why to use them. Parameter descriptions exist but are often terse (e.g., 'Path to X JSON file' repeated 7 times). No tool declares whether it reads or writes beyond the 'Risk' annotation in metadata. Critical issue: tool names do not clearly distinguish operations on the same resource (e.g., 'challenges' loads, 'challenge_start' starts, 'challenge_submit' submits, verb consistency weak). Descriptions lack LLM-optimized context: 'Loads challenge definitions from .checkyourself/challenges.json' tells WHERE but not WHY or WHEN to call it.
Loads and ranks findings from a score result, applies workflow disposition and deferral rules, and emits a prioritized backlog with learning-plan integration.
Emits the server's own tool descriptions, input/output schemas, and MCP protocol metadata.
Starts a challenge by ID and emits a receipt with HMAC binding and timestamp.
Submits a challenge result JSON and validates it against the challenge-result schema before writing to .checkyourself/challenge-runs/.
Loads challenge definitions from .checkyourself/challenges.json and emits a ranked list of active challenges with HMAC receipts.
Validates completion-evidence JSON against the checkyourself-coverage/1 schema and emits a structured coverage report with surface/evidence assessment.
Loads scan, coverage, and score JSON and emits a unified dashboard-data object suitable for rendering in web or markdown templates.
NO tool declares output schemas or return types. Agents cannot predict response structure without trying the tool. Every tool needs documented return fields (e.g., 'Returns: {findings: [{risk, path, detail}], total_count: int, scan_time_ms: int}').
Parameter descriptions are repetitive and lack context. Seven tools say 'Path to X JSON file' without explaining what field to extract, when to call it, or what happens if the file doesn't exist. Descriptions should guide LLM selection ('Use after scan() completes to validate coverage JSON against schema') rather than just name the input.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 63 | 2026-07-28+ | v2 |
Loads a backlog and emits a recommended batch of findings to work on next, bounded by effort estimate and time budget.
Validates a challenge receipt JSON and verifies its HMAC signature using a local integrity binding key.
Discovers cheap local facts about the codebase: stack, dependencies, test coverage markers, known risks, secrets, debuggable paths, CORS misconfiguration, dangerous sinks, default credentials, source maps, and lockfile presence.
Scores finding/coverage JSON against bounded rubric and emits a workflow-aware confidence score, rank order, and next-action recommendation.
Tool naming inconsistency in challenge workflow. 'challenges' (load list), 'challenge_start' (start one), 'challenge_submit' (submit result). Should be 'list_challenges', 'start_challenge', 'submit_challenge' for parallelism. Verb-first naming makes the tool suite scannable and reduces LLM confusion when choosing among similar operations.
Tool 'capabilities' has ambiguous name and purpose. The description 'Emits the server's own tool descriptions, input/output schemas, and MCP protocol metadata' is self-referential and tells agents nothing about WHEN to call it. Should be 'describe_capabilities' or 'get_server_capabilities' and describe the use case (e.g., 'Call to introspect available audit operations before workflow planning').
No error handling guidance in any tool description. If 'score' fails because findings_path doesn't exist, the agent has no recovery hint. Should add: 'Error recovery: if path not found, call scan() first to generate findings JSON.'
Parameter 'exclude_dirs' in 'scan' tool uses array of strings but lacks enum constraint or example. Should clarify format: 'Array of glob patterns (e.g., ["node_modules", ".*", "vendor"]) to skip during traversal. Defaults to ["node_modules", "dist", ".git", ...].'
Parameters 'batch_size' and 'time_budget_minutes' in 'next_batch' lack min/max constraints. Should declare: 'batch_size: integer, range 1-100, default 10' and 'time_budget_minutes: integer, range 1-480, default 60'. Unbounded numbers invite nonsensical LLM inputs.