Loading ToolBench
Model Context Protocol (MCP) client + Tool adapter for Forge. Enables Forge agents to discover and call tools exposed by MCP servers.
Forge MCP defines 9 tools with basic schemas and descriptions, but exhibits significant quality gaps. All tools have descriptions (10-200 chars, within baseline range) and JSON Schema input definitions with proper types and enums where applicable. However, parameter descriptions are sparse or missing entirely, most params lack explanatory text beyond the name. Output schemas are undocumented; the rubric baseline shows 100% of A+ tools document return types, but Forge provides no structured output documentation. Error handling is minimal; tools return anyhow::Result with no recovery guidance. Tool naming is verb-forward (echo, reverse, read_file, search_text) but some names are generic (echo, reverse) and lack specificity about use cases. No tool annotations (readOnlyHint, destructiveHint, idempotentHint) are present despite all tools being READ_ONLY. The run_tests tool is particularly concerning, it executes arbitrary Rust test suites with a 90-second timeout and truncates output at ~8KB, creating silent failures and no guidance on partial results.
Performs arithmetic on two numbers. Input: { op, a, b } where op is one of add, sub, mul, div.
Counts newline-terminated lines in a text file. Input: { path }. Returns { lines, bytes }.
Returns the input string unchanged.
Lists files and subdirectories in a directory. Input: { path }. Returns array of { name, is_dir, size_bytes }, sorted by name.
Reads a file's contents as UTF-8 text. Input: { path, max_bytes? }. Returns { content, bytes_read, truncated }. Default cap is 65_536 bytes.
Returns the input string reversed.
Runs `cargo test` on a Rust project. Returns { exit_code, passed, summary, stdout, stderr }. Input: { manifest_path: string, test_filter?: string, timeout_secs?: integer }. Default timeout 90s. Outputs are truncated at ~8KB.
No output schemas documented. Rubric baseline: 100% of A+ tools document return types. Agents cannot plan downstream calls or extract results without knowing response structure.
Parameter descriptions missing or minimal. Rubric requires every parameter to have a non-empty description explaining what it controls. E.g., 'message' in echo/reverse lacks context; 'path' in list_files/read_file/count_lines/search_text lacks format/constraint guidance.
run_tests truncates output at ~8KB with no indication of truncation or guidance on partial results. Rubric: when multi-step operations partially fail, provide compensation tools or clear error guidance. Silent truncation leaves agents unaware of incomplete test results.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 54 | 2026-07-28+ | v2 |
Searches a text file for lines containing a substring. Input: { path, query, max_matches? }. Returns { matches: [{line, text}], count }. Default max_matches is 50.
Returns the whitespace-separated word count of the input.
Generic tool names (echo, reverse) lack specificity about use cases. Rubric: tool name alone should convey what happens. 'echo' and 'reverse' are too generic for LLM disambiguation when many tools are available.
No tool annotations present. All 9 tools are READ_ONLY but lack readOnlyHint annotation. Rubric: tool annotations (readOnlyHint/destructiveHint/idempotentHint) enable agents to reason about safety and retry semantics.