A benchmarking framework for testing AI agents against MCP servers and CLI tools. Supports multiple transport types (stdio, HTTP, SSE) and CLI command wrapping.
The server exposes only one tool: 'execute'. This tool has a critically poor definition. The description ('Execute CLI command with arguments') is minimal (36 chars, below the 50-200 char baseline for LLM-optimized descriptions). The input schema declares only a single 'args' parameter as a flat string type with minimal description ('Command arguments to execute'), offering no validation, no constraints, and no guidance on format or safety. The schema lacks any output documentation, callers have no idea what the tool returns. The tool name 'execute' is generic and does not follow verb_noun convention (e.g., 'run_command' would be clearer). Most critically, this is a WRITE-risk tool (arbitrary CLI execution) with NO error handling documentation, NO permission gates, NO audit trail guidance, and NO safety constraints. The rubric explicitly requires destructive/sensitive tools to declare permissions, log usage, sanitize input against injection, and provide actionable error messages. This tool violates all of those. No output schema is documented. The server is fundamentally unsafe for agent use without major hardening.
Execute CLI command with arguments. Wrapper tool that allows testing CLI-based tools (like excel-cli) using the agent-benchmark framework.
Generic tool name 'execute' does not convey intent. Should use verb_noun pattern like 'run_command' or 'execute_shell_command'.
Tool description is 36 characters, below the 50-200 char LLM-optimized baseline. Lacks WHAT it does, WHEN to use it, and prerequisites. Does not state that it modifies state (WRITE risk).
Input parameter 'args' has no type constraints, format specification, or examples. Single string parameter invites arbitrary CLI injection. No enum, regex, or length bounds documented.
No output schema documented. Callers cannot plan downstream usage or extract structured data. Violates pattern:tool requirement.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 14 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 15 | - | v1 |
Destructive/sensitive tool (WRITE risk, arbitrary command execution) lacks permission gates, scope declarations, and audit trail guidance. No mention of security model, user authorization, or logging.
No error handling guidance. Tool provides no recovery instructions for failure cases (timeout, permission denied, command not found, injection detection). Violates pattern:recovery-guide.
No input sanitization or injection prevention documented. Tool accepts arbitrary CLI arguments with no mention of SQL injection, command injection, or path traversal guards.
Tool does not declare required permissions (e.g., 'execute:system', 'write:filesystem'). Agents cannot be configured with least-privilege access.