Loading ToolBench
Typed, sandboxed Code Mode access to configured MCP servers for Pi
pi-codemcp presents a sophisticated meta-tool architecture for discovering and executing MCP capabilities. Tool definitions are well-structured with detailed schemas and descriptions (avg 150-250 chars), exceeding baseline minimums. However, several tools exhibit vague or overly complex descriptions that obscure their primary purpose for LLM selection. Parameter schemas are comprehensive with proper type constraints, enums, and limits. Error handling guidance is minimal, tools lack recovery hints. The sandbox execution model (codemcp_execute, codemcp_edit_execute) introduces novel composition patterns but documentation does not adequately explain failure modes or retry semantics. Output schemas are partially documented but inconsistent across tools.
Edit and re-execute the previous codemcp_execute code. Replaces exact text in the previous execution code and runs the modified version in the same sandbox context.
Execute sandboxed Python code with typed access to MCP tools. Call typed SDK methods such as await linear.list_issues(arguments) and return a compact final value. Code runs in an isolated sandbox with access to input reference data and MCP tool SDK methods.
Load full type stubs for selected MCP tools. Returns prelude with imports and type definitions, plus individual tool stubs with complete signatures and docstrings. Use after codemcp_search to inspect selected tools before writing codemcp_execute code.
Manage saved chains: list, enable, disable, revalidate, or delete. Chains can be scoped to project or global storage. Revalidate checks chain syntax and schema compatibility.
Use Jev to select configured MCP calls for the agent's current subtask, including prerequisites, classify each call's workflow role, recommend parallel or dependent composition, and return exact typed SDK contracts. Use when the subtask may require external services or saved workflows. If no configured capability applies, returns no calls.
codemcp_route description is vague and domain-specific ('Use Jev to select...'), LLMs cannot infer when to call this vs codemcp_search. Lacks clear WHEN/WHY guidance.
codemcp_execute and codemcp_edit_execute lack error recovery guidance. No description of what happens on Python syntax errors, timeout, or sandbox escape attempts. LLMs cannot self-correct.
Output schemas are not documented for any tool. LLMs cannot predict what fields to expect from codemcp_search results, codemcp_inspect stubs, or codemcp_route recommendations. Breaks downstream composition.
codemcp_edit_execute requires 'exact text match' but does not explain what happens if oldText appears multiple times or not at all. Ambiguous failure mode.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | 2025-06-18+ | v2 |
Save a reusable native tool (saved chain) from sandboxed Python code. The chain becomes available as a typed SDK method in codemcp_execute and codemcp_route. Chains can call other MCP tools and chains, and are stored in project or global scope.
Search configured MCP capabilities or page through compact inventory. Default signature search includes exact stubs for up to three top matches plus compact alternatives; codemcp_inspect loads other selected stubs. Unscoped searches return available results plus explicit discovery failures; scoped searches remain fail-fast. Returns ranking evidence, pagination, scope, and execution limits; invalid server names fail with suggestions.
codemcp_save_chain and codemcp_manage_chains accept 'scope' (project|global) but do not document scope resolution order, visibility, or conflict handling when chains exist in both scopes.