Loading ToolBench
Recursively searches markdown files in a specified directory based on keywords, prioritizing README.md.
Single tool with basic schema and description. Tool name 'search_markdown' follows verb_noun convention (+). Description is present but generic (58 chars, below 194-char baseline). Input schema properly typed with required array, but lacks output schema documentation (critical gap). No parameter descriptions beyond the query field itself. No error handling guidance, no pagination support despite potentially returning large result sets. No tool annotations. Composition is simple but functional.
Search markdown files recursively in the configured directory by keyword. README.md is prioritized.
Missing output schema documentation. Tool returns text content but LLM cannot infer structure of response (file names, match counts, content format). LLMs need to know what fields to expect.'
No pagination or result limiting. Tool recursively searches all markdown files and concatenates full content into single response. Large codebases could return megabytes of text, exhausting context window. Rubric baseline: 'cap results at 20-50 and offer pagination.'
Generic tool description (58 chars). Does not explain WHEN to use this tool vs alternatives, what prerequisites exist (MARKDOWN_DIR env var), or what happens on no matches. Rubric: 'Write descriptions as if prompt-engineering. State WHAT, WHEN, and prerequisites.'
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 44 | 2026-07-28+ | v2 |
No error recovery guidance. Error message 'An error occurred while searching markdown files' is opaque. LLM cannot determine if error is retryable, user-fixable, or fatal. Rubric: 'Error responses must tell the LLM what to do next.'
No tool annotations. Tool is read-only but lacks readOnlyHint annotation. Per current spec (2026-07-28), tool annotations enable client-side safety checks and UI hints.