Loading ToolBench
The mcp-gateway server exposes 3 filesystem tools (read_file, write_file, list_directory) with basic but incomplete definitions. All tools have names and descriptions present, but parameter and output documentation is severely deficient. Schemas are visible but lack completeness: parameters have type and description fields, but there is no documented output schema for any tool. Descriptions are minimal (10-50 chars) and lack context about when to use each tool or what to expect in return. Error handling is not visible in the tool definitions. The server uses STDIO transport, which is a hard cap on protocol readiness. Tool names follow verb_noun convention (read_file, write_file, list_directory), which is a positive, but descriptions do not explain side effects, return types, or error cases.
list entries in a directory with type indicators
read and return the contents of a file
create or overwrite a file with the given content
Output schemas not documented. No tool specifies what fields it returns, their types, or structure. LLMs cannot infer downstream tool compatibility or plan multi-step operations.
Descriptions are too brief (10-50 characters) and lack context. 'read and return the contents of a file' does not explain: What happens if the file doesn't exist? What's the max file size? What formats are supported? When should this tool be used vs a URL fetch?
write_file description does not state that it is destructive (overwrites existing files). Agents need to know which calls have irreversible consequences. Recommendation: 'Creates or overwrites a file. WARNING: Overwrites existing files without confirmation.'
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | 2026-07-28+ | v2 |
No error handling guidance. If read_file is called on a non-existent file, the tool response provides no recovery hint (e.g., 'Try list_directory() to see available files'). Error responses should guide the LLM's next action.
No parameter constraints documented. 'path' parameter has no validation hints (e.g., 'must be absolute', 'no parent directory traversal allowed', 'max length 512 chars'). LLMs lack guidance on valid inputs.
No explicit path traversal or sandbox constraints documented. For a filesystem tool, the description must state: 'Paths are restricted to <sandbox_dir>. Parent directory references (..) are blocked.' This prevents agents from accidentally exfiltrating sensitive files.