Loading ToolBench
MCP server for secure command execution with full security and audit capabilities
Two tools with mixed quality. Both have descriptions and basic schemas, but parameter descriptions are minimal/generic, output schemas are not documented, and error handling lacks actionable guidance. The naming follows verb_noun convention (execute_command, check_command_allowed), which is good. However, execution_command's description is vague about what 'full security checks' entails, and parameter descriptions do not explain constraints, ranges, or validation rules. Parameter descriptions are extremely brief (5-15 chars) and fail to meet the 10-1024 char guideline for tool descriptions or the requirement that every parameter have meaningful documentation. Output types are not explicitly documented. Error handling in the code shows custom exceptions (ValidationError, PermissionError) but the tool definitions do not guide LLMs on recovery. The schema definitions visible in the code show proper JSON types (string, array), but descriptions are missing the constraint/format detail needed for agent reasoning.
Check if a command would be allowed without executing it.
Execute a system command with full security checks.
Parameter descriptions are trivial (5-15 characters). Rubric requires 10-1024 chars and meaningful explanation of what each parameter controls. 'Command arguments' and 'Working directory' do not explain constraints, format expectations, or valid ranges.
Output schemas are not documented. Rubric §D requires documentation of what fields to expect in the response. execute_command returns a string (command output), but what structure? Is it raw stdout, JSON with stdout/stderr split, or wrapped in a status object? This forces LLMs to guess.
Error messages lack recovery guidance. Code raises generic ValidationError and PermissionError, but tool definitions do not explain what LLM should do next (e.g., 'Try check_command_allowed first', 'Ask user for confirmation'). Rubric §E requires 'Error responses must tell the LLM what to do next.'
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 41 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 35 | - | v1 |
No tool annotation hints. Both tools are command-execution (destructive/write) or read-only, but no readOnlyHint, destructiveHint, or idempotentHint annotations are visible. Agents cannot reason about side effects.
execute_command description states 'full security checks' but does not enumerate them (validation, sanitization, confirmation, audit). Rubric §B requires descriptions to state WHAT, WHEN, and prerequisites. This is too vague for agent planning.
No input validation constraints visible in parameter descriptions. The 'command' parameter accepts any string, but the code enforces an allowlist (DEFAULT_ALLOWED_COMMANDS). Rubric §C requires description to state constraints: 'Must be one of: ls, cd, npm, python, pip, git'. Without this, LLMs hallucinate invalid commands.
check_command_allowed returns a boolean, but no documentation of what true/false means or how it differs from execute_command. Should an LLM call this first? When? The relationship between the two tools is unexplained.