Loading ToolBench
MCP server for executing SWE-Bench tasks with autonomous repository refactoring, debugging, and code modification capabilities
Critical failures across all dimensions. The single tool 'execute_skill_action' has a minimal description (89 chars, below the 194-char baseline), lacks parameter descriptions entirely, and has no documented output schema. The mcp_server.py implementation is a stub that prints JSON and exits, it does not implement the MCP protocol at all. Tool definitions are inferred from client.py example code, not explicitly registered. No error handling, no validation, no security controls.
Execute SWE-Bench task with autonomous repository refactoring and debugging
Tool description is too short (89 chars vs 194-char baseline) and lacks context for LLM selection. Does not explain WHEN to use it, what it returns, or how it differs from similar tools.
Input parameters 'github_issue_prompt' and 'repo_url' have no descriptions. LLMs cannot infer parameter meaning from names alone, a 'github_issue_prompt' could be a title, full issue text, or structured JSON.
No output schema documented. The client.py example shows a response with 8 fields (task_session_id, target_repository, files_modified_count, etc.), but the tool definition does not declare this structure. LLMs cannot plan downstream calls without knowing what fields to expect.
Tool name 'execute_skill_action' is vague and generic. Does not start with a clear action verb (e.g., 'refactor_repository', 'debug_swe_issue'). LLMs cannot infer intent from the name alone.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 24 | 2026-07-28+ | v2 |
No error handling or recovery guidance. If the tool fails (invalid repo URL, network error, SWE-Bench timeout), the response provides no actionable next steps for the LLM.