Loading ToolBench
Reusable Model Context Protocol server framework + 3 example servers (filesystem-stats, github-issues, sqlite-query). In-memory test harness with sub-millisecond tool round-trip.
9 tools with clear naming (verb_noun pattern) and reasonable descriptions (avg ~120 chars). All tools have input schemas with type definitions. However, parameter descriptions are sparse or missing entirely (e.g., 'rel_path' in list_directory lacks detail on what 'sandbox root' means). Output schemas are undocumented, LLMs cannot predict return structure. Error handling is minimal; no recovery guidance. Security is reasonable (read-only operations, no secrets exposed) but lacks explicit permission declarations. Composition is sound, tools are single-purpose and chainable.
Return columns + types for a single table.
Return size, suffix, and (for text files <= 1 MB) line count for one file under the sandbox root.
Find files matching a glob pattern under the sandbox root. ``max_depth`` caps recursion; ``limit`` caps the number of returned paths.
Get one issue's title, state, body, and first 5 comments.
List entries of a directory under the sandbox root. Returns name, type, size_bytes, and modified time for each entry. ``rel_path`` is relative to the sandbox root.
List open issues on a repo, newest first.
Output schemas undocumented. LLMs cannot predict return structure (e.g., what fields does list_directory return? Is it a list of objects or a dict?). This forces agents to guess and wastes tokens on trial-and-error parsing.
Parameter descriptions are minimal or missing context. 'rel_path' lacks explanation of what 'sandbox root' means or how paths are resolved. 'pattern' in find_files does not explain glob syntax or escaping rules. LLMs cannot infer constraints from names alone.
No error recovery guidance. Tools lack descriptions of failure modes (e.g., what if a file is not found? What if a GitHub API rate limit is hit?). Agents receive errors but no actionable next steps.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 66 | 2026-07-28+ | v2 |
List user tables in the configured SQLite database.
Run a read-only SQL query (SELECT or WITH ... SELECT). Returns up to ``100`` rows as TSV. Write statements are rejected.
Search GitHub issues across all public repos using GitHub's query syntax. Returns up to ``limit`` results, newest first.
No pagination or result limits documented in descriptions. 'search_issues' returns 'up to limit results' but does not explain what happens if limit is exceeded or how to fetch the next page. Large result sets risk context window exhaustion.
No permission declarations. Tools are read-only but lack explicit scope annotations (e.g., 'read:filesystem', 'read:github'). This prevents least-privilege agent configuration and audit clarity.