Loading ToolBench
Mechanical QA for ink stories — compile checks, bounded branch exploration, repro paths, and dead-content review.
Inkcheck exposes 7 tools with reasonable naming (verb-prefixed) and detailed descriptions averaging ~250 chars, but exhibits critical schema and composition gaps. Tool definitions are visible in src/server.ts with explicit registration, but JSON Schema input validation is not shown in the provided code excerpt. Descriptions are generally substantive and operational (e.g., 'pin one confirmed runtime error...'), addressing WHEN to use each tool, but schema output documentation is absent. Parameters are typed in descriptions but lack formal JSON Schema constraints (enums, ranges, format). Error handling is not visible in the provided code. The server's domain (ink story testing) is specialized and the tool names map clearly to domain concepts, but several tools conflate multiple concerns (e.g., 'inkcheck_workflow' is a catch-all dispatcher, violating single-responsibility principle). No tool annotations (readOnlyHint, destructiveHint, idempotentHint) are evident.
After editing current source, replay one private evidence pin without spending search states. Runtime and assertion pins return fixed, still_failing, or path_changed; goal witness pins return still_reached, lost, or path_changed. This rechecks one exact witness, not broad coverage.
Compile an ink story and return issues, word count, and choice count. This checks syntax, includes, and externals without running exploration.
Return Inkcheck's versioned schemas, limits, search modes, and explicit supported/unsupported feature flags. Call this before relying on optional functionality.
Execute a named workflow operation: review_contract, inspect_search, continue_search, start_campaign, continue_campaign, get_finding, open_report, replay_witness, pin_regression, check_regression, add_goal, probe_gate, add_assertions, cancel_search, or playtest_story.
Discover knots, choices, variables, and goals in the compiled story without exploration. Returns inventory, entrypoint, and schema version.
CRITICAL: inkcheck_workflow tool is a catch-all dispatcher that conflates 15+ distinct operations (review_contract, inspect_search, continue_search, start_campaign, continue_campaign, get_finding, open_report, replay_witness, pin_regression, check_regression, add_goal, probe_gate, add_assertions, cancel_search, playtest_story) into a single tool. This violates the single-responsibility principle and forces the LLM to reason about which operation to select and guess the 'arguments' schema for each variant. Proper design would create separate tools: inspect_search, continue_search, start_campaign, continue_campaign, get_finding, open_report, replay_witness, add_goal, probe_gate, add_assertions, cancel_search, playtest_story.
Output schemas are not documented for any tool. Description text mentions return values in prose (e.g., 'Returns inventory, entrypoint, and schema version' for inspect_story) but no formal JSON Schema or structured definition is provided in the code. LLMs cannot plan downstream tool calls or validate responses without knowing the output shape. Required: add explicit output schema documentation (JSON Schema format) to each tool definition.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 54 | 2026-07-28+ | v2 |
Pin one confirmed runtime error, assertion violation, or goal witness from the latest current report for deterministic post-edit checks. The private bounded artifact stores indexed choices, story seed, and typed rule data without story prose or observed variable values; a pin checks one witness, not the entire story-wide result.
Start a bounded exploration session. Returns a session capability and initial search revision. Use continue_search or start_campaign to advance.
Input parameter schemas lack formal constraints (enums, ranges, patterns, formats). For example, 'search' parameter in start_search lists valid values (shared, dfs, beam, random) in the description but does not declare an enum constraint in JSON Schema. 'maxStates' and 'maxDepth' accept integers but have no documented min/max bounds. 'reportId' is optional but no guidance on when to include it. Formal schemas (JSON Schema with type, enum, minimum, maximum, pattern) are machine-parseable and prevent hallucinated values.
Error handling and recovery guidance not visible. No tool definition shows error responses, categorization (retryable vs fatal), or actionable guidance for LLMs when calls fail. E.g., if start_search fails because maxStates exceeds a limit, what should the LLM try next? Without explicit error contracts, agents will retry blindly or give up.
Tool naming ambiguity for domain-specific terms: 'pin_regression' and 'check_regression' are jargon-heavy and may not clearly communicate their purpose to LLMs unfamiliar with testing terminology. 'pin' as a verb is not standard; 'create_regression_checkpoint' or 'snapshot_regression_state' would be clearer. 'check' vs 'validate' vs 'verify' adds cognitive load.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) present in tool definitions. All tools appear to be READ_ONLY based on the 'Risk' field in the provided spec, but this is not declared in the MCP protocol. Modern MCP requires tools to annotate their safety properties so agents and frontends can make informed decisions about retries and logging.