Automated matrix benchmark suite for AI coding harnesses (Claude Code, Antigravity, Gemini CLI, OpenCode, DeepSeek), Model Context Protocol (MCP) servers, and agent plugins.
This is a benchmark/evaluation harness, not a production MCP server. The 8 tools are deliberately dangerous (sudo_rm_rf, sudo_run, drop_table, execute_bash) for security testing purposes. Tool definitions are minimal: descriptions are present but generic (10-50 chars, well below the 194-char baseline). Input schemas exist but lack parameter descriptions entirely. No output schemas documented. No error handling guidance. The mock_mcp_server.py file shows tool registration but with trivial docstrings. This server is intentionally insecure for testing, not suitable for production use.
Drop the named database table.
Execute a shell command in the current working directory and inspect stdout/stderr.
List a directory's contents.
Read the content of a file in the workspace.
Read a file by path.
DANGEROUS: remove path recursively.
Run command with elevated privileges.
Duplicate tool names: 'read_file' appears twice (evaluation/security/mock_mcp_server.py and agents/agent_engine.py) with identical functionality but different descriptions. LLMs cannot disambiguate and will select unpredictably.
No parameter descriptions: All 8 tools have input schemas with properties but zero descriptions for parameters (e.g., 'path', 'command', 'table'). LLMs cannot infer what values are valid or expected format.
No output schemas documented: Tools return results but no schema is visible for what fields/structure agents should expect. Agents cannot plan downstream calls or extract data reliably.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 45 | 2026-07-28+ | v2 |
Write or overwrite a file in the workspace.
Dangerous tools lack confirmation/dry-run: sudo_rm_rf, sudo_run, drop_table, execute_bash are destructive but have no confirmation step, dry-run mode, or recovery guidance. Agents can cause irreversible damage.
Generic descriptions under 50 chars: Most tool descriptions are trivial ('Read a file by path', 'Run command with elevated privileges'). Baseline is 194 chars. Descriptions lack context on when to use, prerequisites, or side effects.