Loading ToolBench
AI-powered command-line assistant with MCP integration providing search, CLI execution, file listing, and vector database memory management capabilities
This server has 4 tools with significant quality gaps. Tool naming follows verb_noun convention (positive), but descriptions lack context for LLM selection and parameter constraints. Schemas are present but incomplete, they lack enum constraints, range documentation, and output structure definitions. Critical security issues: execute_cli_command exposes destructive capabilities with only basic string blacklisting (not sufficient); get_memory stores user data without documented access controls. Error handling exists but is generic (raw exception strings). Most tools return unstructured text instead of machine-parseable JSON. Baseline for median production tool: 194 chars description, 72 chars per param, this server averages ~150 chars for tool descriptions but 0 chars for param descriptions (relying entirely on param names, which is insufficient).
Execute a command line interface (CLI) command and return its output.
Manage user memory using the vector database.
Perform a Google search and return relevant results.
List files and directories in a specified path.
Parameter descriptions are missing entirely. Tool inputs have JSON Schema types (string, integer, boolean) but zero description text for parameters. LLMs cannot infer intent from bare 'action' or 'timeout' names. Rubric baseline: 100% of A+ tools have param descriptions (average 72 chars each). This server has 0.
Tool descriptions lack actionable context for selection. 'Perform a Google search and return relevant results' (50 chars) does not explain WHEN to use it instead of a similar tool, what prerequisites exist (API keys must be configured), or that it requires environment variables. Baseline: 194 chars average; this is 50 chars.
No output schema documented. All tools return unstructured text strings (e.g. google_search returns formatted text with emoji). LLMs cannot parse or chain results, they must rely on text pattern matching. Rubric: 100% of A+ tools have documented return types. This server has 0 return type documentation.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 48 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 34 | - | v1 |
execute_cli_command: Destructive operations use only string blacklisting, which is fragile (e.g. 'rm' spelled differently, quoted, or obfuscated bypasses the filter). Command execution with user-controllable input is a critical injection vector. Rubric: 'Treat all agent-provided input as untrusted. Sanitize against command injection.' Uses subprocess correctly (shell=False) but semantic validation is insufficient.
get_memory: Tool performs WRITE operations (saves user memory to vector DB) but lacks permission checks, audit logging, and documented access control. Rubric: 'Gate destructive or sensitive tools behind permission checks' and 'Log who called what tool, with which parameters, at what time, and what happened.' No evidence of either.
Parameters lack constraints. 'timeout' has no range bounds (LLM could pass 999999 seconds, causing hangs). 'command' has no length limit or pattern. 'action' in get_memory is a free string ('save' or 'load') but not declared as an enum. Rubric: 'When a parameter accepts one of a known set of values, declare it as an enum.' Enums are self-documenting and prevent hallucinated values.
get_memory parameter 'text' is ambiguous and overloaded: it accepts memory text to save OR a query to search, depending on 'action' value. Rubric: 'When one parameter's valid values depend on another, document this in both parameter descriptions. Undocumented dependencies cause silent misuse.' No documentation of this dependency; LLM must infer from context.
Error handling returns raw exception strings ('Error performing Google search: <exception text>') instead of actionable guidance. Rubric: 'Error responses must tell the LLM what to do next.' No recovery suggestions, error classification, or invocation of related tools (e.g. 'API keys not configured. Try setting GOOGLE_API_KEY in .env'). Missing retry guidance for timeouts.
google_search: Sensitive credentials (GOOGLE_API_KEY, GOOGLE_SEARCH_ENGINE_ID) are correctly loaded from environment variables, not tool parameters (good). However, error message returns 'Google API keys not configured' without guiding the user on how to set them. This is a minor UX gap but correct security practice.
Tool composition: get_memory writes to a vector database but there is no corresponding retrieval or query tool visible in this export. If save and load are meant to be called separately, the 'action' parameter design is awkward (overloaded semantic). If get_memory is the only interface, it violates single-responsibility principle: one tool doing lookup AND storage.