Loading ToolBench
A reference corpus and approval framework for drafting agents, with tools for validating local references, receipts, and managing calibration examples.
Chalkline provides 4 tools with clear, domain-specific purposes (corpus validation, indexing, search, receipt validation). All tools have descriptions (avg 180 chars, within baseline 194). Input schemas are present with typed parameters and descriptions. However, output schemas are not documented, LLMs cannot predict return structure. Parameter descriptions lack format constraints (e.g., search query syntax, JSON schema version expectations). Error handling is minimal; tools raise CorpusError but recovery guidance is absent. No tool annotations (readOnlyHint, etc.). Naming is verb-forward (check, index, search, validate) and unambiguous. Tools are single-responsibility and composable.
Validates structure of local reference corpus; checks that all required frontmatter fields are present and correctly formatted, enforces domain/enforcement constraints, and validates UTF-8 encoding and file structure.
Indexes all Markdown reference files in references/ directory, extracting metadata (title, brand, domain, enforcement, teaching) and computing SHA-256 hashes; outputs JSON manifest sorted by path.
Searches indexed reference corpus using case-insensitive literal whitespace-separated terms (AND logic) across title and body, with optional exact metadata filters (domain, enforcement, brand); returns matching records with line numbers.
Validates structure of a local JSON receipt file, checking schema version, required fields, reference integrity, score arithmetic (0-3 or null), and blocker/evidence constraints; computes and verifies overall score; emits validated JSON to stdout without writing files.
Output schemas not documented. LLMs cannot predict return structure (e.g., search returns 'matching records with line numbers' but field names, types, and pagination are not specified).
Parameter descriptions lack format constraints. 'query' param in search tool does not specify whitespace-separated AND logic syntax, case-insensitivity rules, or escaping. LLMs will guess.
No error recovery guidance. CorpusError exceptions are raised but tool descriptions do not explain what to do if validation fails (e.g., 'If check fails, review frontmatter format in references/ directory').
No tool annotations. Tools are read-only but lack readOnlyHint annotation. Agents cannot infer safety/idempotency from tool definitions alone.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 62 | 2026-07-28+ | v2 |
index tool 'output' parameter lacks validation guidance. Description says 'never overwrites; parents must exist' but does not specify what error is raised if parents don't exist or file exists.