Loading ToolBench
MCP server that runs Python in an embedded CPython interpreter
Strong foundation with well-structured schemas and comprehensive parameter descriptions. All 7 tools have explicit input schemas with type definitions and detailed descriptions (50-300+ chars). Tool names follow verb_noun convention (python_exec, python_check, python_reset). However, output schemas are not documented, LLMs cannot predict return structure. Error handling guidance is minimal. Tool descriptions are good but lack explicit WHEN/WHY context for selection. No tool annotations (readOnlyHint/destructiveHint) despite clear risk levels.
Parse Python source to check for syntax errors. Returns a summary when the program compiles, or the syntax error if it doesn't.
Run Python source code through an embedded interpreter. Hand it a program, get back stdout/stderr/result — no `python -c` framing. Can be multiple statements. If the last statement is a bare expression (like `2 + 2` or `some_var`), its repr is reported the way a REPL cell reports it: `2 + 2` gives `=> 4`. An assignment reports nothing, and neither does an expression whose value is None.
Read the result of a background job started with `python_exec` and `background: true`.
Stop a background job before its budget runs out.
List all background jobs and their status.
Empty a named session. Its variables, imports, and function defs are dropped; the name stays valid, and its next call starts empty.
Output schemas not documented. LLMs cannot predict return structure (e.g., what fields python_exec returns, what python_sessions returns). This forces agents to guess and wastes tokens on trial-and-error.
Tool annotations missing. python_exec and python_job_cancel are destructive (WRITE risk), python_check and python_sessions are read-only (READ_ONLY risk). Annotations (readOnlyHint, destructiveHint) enable agents to reason about safety and retry logic.
Error handling lacks recovery guidance. No indication of what to do if timeout occurs, if a session is blocked, or if a job fails. Errors should guide the LLM: 'ExecutionTimeout, increase timeout_secs or break the program into smaller steps.'
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 59 | 2026-07-28+ | v2 |
List all named sessions and their state.
Tool descriptions lack WHEN/WHY context. python_sessions says 'List all named sessions and their state' but does not explain when an agent should call it (e.g., 'Call this to check if a session is blocked by a long-running job'). This reduces discoverability.
Parameter 'timeout_secs' lacks explicit bounds. Description says 'no upper limit' but does not specify minimum (presumably 1). Unbounded parameters invite absurd values (e.g., timeout_secs=999999999). Add min/max constraints.