Loading ToolBench
MCP server that bridges CLI tools into Claude via structured specs, allowing AI agents to execute command-line tools with purpose-built domains and structured output
The cli-bridge server dynamically generates tool definitions from CLI specifications. However, the evaluation reveals critical gaps in definition quality that severely limit its production readiness. The single tool pattern '{binary}_{subcommand}' is generated at runtime from external spec files (not visible in the provided source code). While the server.ts shows schema building logic using Zod, the actual tool definitions depend entirely on external spec files whose structure and quality are unknown. The tool naming follows a mechanical binary_subcommand pattern that lacks action verbs, 'docker_build' happens to work, but 'docker_info' is ambiguous (get info? describe?). The inputSchema is constructed from spec properties, but there is no evidence of enum validation, required/optional clarity in descriptions, or format specifications (regex, ranges, date formats). Error handling returns exit codes and stderr but provides no actionable recovery guidance, 'exit code 1' tells the LLM nothing about whether to retry, ask the user, or give up. The tool definitions are inferred from runtime spec loading rather than explicitly registered with full schemas visible in code, triggering the 'cannot see actual tool definition' cap.
Dynamic tools generated from CLI tool specs. Tool names are constructed from binary name and subcommand name. Each tool executes the corresponding CLI command with user-provided parameters (flags, arguments) and returns structured output (JSON, JSONL, CSV, TSV, or text).
Tool definitions are dynamically inferred from external spec files (not visible in source code). The rubric explicitly states tool definitions must be evident in code.
Tool naming does not follow verb_noun convention. The pattern '{binary}_{subcommand}' is mechanical (e.g., 'docker_build', 'docker_info', 'kubectl_get') but lacks action verbs in the tool name itself. This forces LLMs to infer intent from descriptions rather than names. The rubric requires names to start with verbs (get_, create_, update_, delete_, search_, list_, send_) so LLMs can parse intent before reading descriptions.
Parameter descriptions are stated to be 'dynamically generated from spec' but are never shown as validated, formatted, or reviewed. The rubric requires every parameter to have a non-empty description explaining what it controls, with expected format, range, and allowed values. No evidence that generated descriptions meet the 10-1024 character guideline or include constraint details (e.g., 'priority must be low|medium|high|urgent').
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 30 | 2026-07-28+ | v2 |
Tool description in server.ts is a generic template: 'Dynamic tools generated from CLI tool specs... Each tool executes the corresponding CLI command with user-provided parameters... and returns structured output.' This is under 50 characters when removing boilerplate and does not answer WHAT the tool does (depends on the spec), WHEN to use it (no guidance), or WHAT it returns (claims 'structured output' but output parsing is delegated to parseOutput() function with unknown quality). Per the rubric, descriptions must be 10 - 1024 characters and answer these three questions.
Input schemas are built dynamically from spec properties with only basic type mapping (string, number, boolean, enum). There is no validation of schema completeness, no evidence of constraints (minLength, maxLength, pattern, minimum, maximum), and no per-parameter descriptions beyond what the spec provides. The schema construction in server.ts shows only type assignment and optional/.describe() calls, but does not enforce format specifications, ranges, or enum validation against known safe values.
Error handling provides no actionable recovery guidance. When a tool exits with non-zero status, the response returns [exit code: N] and stderr text. The rubric requires error responses to tell the LLM what to do next (e.g., 'User not found. Try search_users() with a partial name.'), categorize errors as retryable/user-fixable/fatal, and include the invalid value and constraint violated. Raw exit codes give agents nothing to act on.
Output parsing is delegated to parseOutput(result.stdout, entry.command.output) with no visibility into output schema structure, field naming, or documentation. The rubric requires output schemas to be documented so LLMs know what fields to expect and can plan downstream tool calls. If the output is unstructured text, LLMs must parse it, error-prone and wasteful.
No evidence of validation against parameter constraints. The rubric states 'Do not assume LLMs will follow constraints. Validate inputs early and return clear error messages.' The server builds Zod schemas but there is no pre-execution validation that e.g., enums are respected, numeric ranges are enforced, or required parameters are present (Zod validation exists but error messages are not shown).
No security checks visible for credential injection, command injection, or path traversal. The rubric requires tool inputs to be sanitized against SQL injection, command injection, and path traversal. The executeTool() function is not shown, so it is unknown whether parameters are quoted, escaped, or validated before passing to the CLI. This is a critical gap for a tool that shells out to arbitrary binaries.
No evidence of idempotence guarantees or duplicate-side-effect prevention. The rubric requires tools to produce the same result on repeated calls with the same input. A tool that runs 'docker rm container_id' twice will fail on the second call. There is no indication of idempotency hints or compensating logic.