Loading ToolBench
Deterministic verification for AI-generated analysis. Exposes andon's verify/inspect/diff tools to an agent runtime over MCP.
Three tools with clear, domain-specific purposes and well-structured schemas. Naming is verb-forward (tool_run, tool_inspect, tool_diff) and unambiguous. Descriptions are substantive (120 - 180 chars) and explain WHAT, WHEN, and WHAT-RETURNS. All parameters have types and descriptions. However, output schemas are not formally documented in the code, only inferred from docstrings. Error handling returns structured dicts with 'error' and 'verdict' fields, but lacks recovery guidance (e.g., 'Try X if this fails'). No parameter constraints (enums, ranges) despite accepting string inputs like tolerance. Missing idempotency/destructiveness annotations.
Diff two workbook versions and classify what changed: a new error cell on its own, numeric moves with a delta and a percent, formula edits, appeared/vanished cells, added/removed sheets. `tolerance` (e.g. "0.5%") hides numeric noise.
Integrity-scan an .xlsx/.xlsm workbook without writing a spec: error cells (#REF! …), values typed over formulas, numbers stored as text, hidden rows, external links. Optionally limit to specific worksheets.
Run an andon verification spec (YAML) and return the full report: verdict, per-check results with evidence, and the honesty block (what was read, what was skipped). The `verdict` field is the go/no-go signal.
Output schemas not formally documented. Docstrings describe return structure (verdict, per-check results, honesty block) but no JSON Schema definition visible in code. LLMs cannot reliably parse or chain results.
No parameter constraints or enums. 'tolerance' accepts free-form strings (e.g., '0.5%') with no validation or format specification. 'sheets' is an array with no length limits. LLMs may pass invalid values.
Error responses lack recovery guidance. Returns {'error': str, 'verdict': 'SPEC_ERROR'} but does not tell LLM what to do next (e.g., 'Check YAML syntax' or 'Verify file paths exist').
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). All three tools are read-only and idempotent, but this is not declared in the schema. Clients cannot optimize caching or retry logic.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 62 | 2026-07-28+ | v2 |
File paths as strings with no validation. 'spec_path', 'workbook_path', 'before', 'after' accept arbitrary strings. No path traversal guards visible; relies on runtime exception handling.