Loading ToolBench
Sandy Play CLI - Execute MCP scenarios without LLM (deterministic replay). Provides MCP client factory with support for multiple transports (stdio, SSE, WebSocket, Unix socket) and native tool implementations.
Sandy provides 8 tools with reasonable schemas and descriptions, but falls short of production-grade quality. All tools have explicit names, descriptions (averaging ~60 chars), and input schemas with typed parameters. However, critical gaps emerge: (1) Tool names lack action verbs, they all start with 'claude__' prefix without clear verb patterns (read, write, edit are verbs, but glob, grep, bash, web_fetch, notebook_edit vary in clarity). (2) Descriptions are minimal (10-60 chars) and do not explain WHEN to use each tool or why one would be chosen over another. (3) Parameter descriptions are present but sparse, most are 1-2 sentences and lack explicit constraints, ranges, or format guidance. (4) Output schemas are NOT documented, tools return dicts with fields like 'content', 'path', 'files', 'count' but no formal schema definition is provided to guide downstream agent reasoning. (5) Error handling is basic, tools raise Python exceptions that are caught and wrapped in ToolResult(success=False, error=str(e)), but error messages do not guide recovery. (6) No tool annotations (readOnlyHint, destructiveHint, idempotentHint) are present, leaving agents to guess which tools are safe to retry. (7) Security: write/edit/bash/notebook_edit are destructive, but no confirmation patterns or dry-run modes exist. claude__bash in particular accepts arbitrary shell commands with no input sanitization or audit trail mentioned. (8) Composition: tools do not validate that output IDs match downstream tool parameter names, e.g. glob returns 'files' list but grep accepts 'path' string, forcing the agent to reason about transformations.
Execute shell commands
Edit file by replacing string
Find files matching glob pattern
Search file contents with regex
Edit Jupyter notebooks
Read file contents
Fetch URL contents
Write content to a file
No output schema documentation. Tools return unstructured dicts (e.g. claude__read returns {'content': ..., 'total_lines': ..., 'start_line': ..., 'lines_read': ...}) but agents cannot inspect schema to extract fields for downstream calls. Agents must infer structure from implementation.
Minimal tool descriptions lack WHEN/WHY guidance. 'Read file contents' does not explain when to call claude__read vs claude__grep. 'Execute shell commands' does not warn agents about security risks or irreversibility. Descriptions average ~50 chars; production baseline is 194 chars.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 41 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 0 | - | v1 |
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Agents cannot determine which tools are safe to retry. claude__write, claude__edit, claude__bash, claude__notebook_edit are all destructive but not marked. Agents may retry them inappropriately, causing duplicate writes or commands.
Error handling is basic. Exceptions are caught and wrapped as ToolResult(success=False, error=str(e)), but error messages do not guide recovery. E.g. claude__bash timeout or command failure returns a raw error string with no retry guidance or alternative suggestions.
claude__bash accepts arbitrary shell commands with no input sanitization mentioned. No audit trail, rate limiting, or permission checks. An agent (or prompt injection attack) could invoke 'rm -rf /' without safeguards.
Parameter descriptions are sparse and lack explicit constraints. E.g. claude__grep 'head_limit' has no range guidance (is 0 valid? 1,000,000?). claude__bash 'timeout' is a number with no min/max (0 seconds? negative?). Agents must guess valid ranges.
No idempotency guarantees. Agents retry on ambiguous failures. claude__write, claude__edit, and claude__bash can produce duplicate side effects if retried (overwrite file twice, execute command twice). No idempotent markers or idempotency keys provided.
Tool names do not follow standard verb_noun pattern consistently. Names like 'glob' and 'grep' are Unix utilities, not action verbs. Agents benefit from explicit verbs: 'search_files' (glob), 'search_content' (grep). Current naming is terse and tool-centric rather than intent-centric.
No confirmation or dry-run pattern for destructive operations. claude__write (overwrites), claude__edit (modifies), claude__bash (irreversible), and claude__notebook_edit all execute immediately without confirmation. Agents can accidentally destroy data.