Loading ToolBench
Claude Code experience for the pi coding agent: reads .claude config (rules, commands, skills, hooks, output styles, MCP servers, agents) and adds todo, checkpoints, memory, web, subagents, and goals
pi-code exposes 16 tools with significant definition gaps. Most tools lack complete input schemas visible in source; descriptions are present but often generic (10-50 chars). Critical issues: (1) No input schemas visible for web_fetch, web_search, question, plan_mode_complete, slash_command, todo, ls, list_mcp_resources, these 8 tools score 0 on schema. (2) Descriptions are too brief: 'Ask the user a question' (27 chars), 'Mark plan mode as complete' (26 chars), 'List directory contents' (22 chars) fall below the 50-char baseline for LLM clarity. (3) Tool names lack verb clarity: 'subagent' is vague (should be 'delegate_to_subagent' or 'run_subagent'); 'slash_command' is generic. (4) Parameter descriptions missing for many tools, grep has '-i' flag but no description; find, question, plan_mode_complete have no visible input specs. (5) No output schemas documented for any tool. (6) Error handling guidance absent, tools do not explain recovery paths. Average tool definition quality is below median (45-55 baseline).
Execute shell commands
Edit file contents with one or more text replacements
Find files matching a pattern
Search for patterns in files
List all available MCP resources from connected servers
List directory contents
Mark plan mode as complete
Ask the user a question
8 of 16 tools have no visible input schemas (web_fetch, web_search, question, plan_mode_complete, slash_command, todo, ls, list_mcp_resources). LLMs cannot infer parameter types or constraints.
Descriptions are too brief (10 - 50 chars) for LLM clarity. 'Ask the user a question' (27 chars), 'List directory contents' (22 chars), 'Mark plan mode as complete' (26 chars) lack context on WHEN to use and WHAT to expect. Baseline is 50 - 200 chars.
Tool names lack action verbs or are vague. 'subagent' should be 'delegate_to_subagent' or 'run_subagent'; 'slash_command' is generic and does not convey what it does. LLMs struggle to infer intent from vague names.
Inferred effective spec: 2025-06-18+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 43 | 2025-06-18+ | v2 |
Read file contents
Read content from an MCP resource
Execute custom slash commands from skills
Delegate tasks to specialized subagents with isolated context. Modes: single (agent + task), parallel (tasks array), chain (sequential with {previous} placeholder). Single mode also supports background: true for long tasks; a notification arrives on completion and {status: true} lists runs. Agents come from ~/.claude/agents and ~/.pi/agent/agents, plus project .claude/agents and .pi/agents once the project is trusted. agentScope: "user" or "project" narrows to one source.
Create, read, update and list todo items and tasks
Fetch content from web URLs
Search the web
Write content to a file
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields (e.g., does 'read' return {content, size, encoding} or just {content}?).
No error handling guidance. Tools do not explain recovery paths (e.g., 'File not found. Try ls() to list available files.' or 'Invalid agent name. Call list_agents() to see available options.').