Loading ToolBench
Memo Code is a terminal-based coding agent with 23 tools spanning shell execution, file I/O, MCP resource interaction, and multi-agent collaboration. Critical assessment reveals pervasive gaps in schema visibility, parameter documentation, and output specification. Of 23 tools examined, input schemas are not visible in the provided source code for ANY tool, only tool names and brief descriptions are present in `packages/core/src/tools/index.ts`. The actual implementation files (e.g., `exec_command.ts`, `write_file.ts`) are referenced but not provided, making it impossible to verify parameter types, descriptions, or output schemas. Per the hard scoring rules: tools without visible input schemas score 0 on schema, and tools without visible definitions are capped at 50 overall. Additionally, all tool descriptions are extremely brief (9-37 characters), well below the 34-character p10 baseline and far below the 50-200 character LLM-optimal range. No tool includes parameter-level descriptions, which is a critical gap per the rubric (100% of A+ tools have all params described). Error handling guidance is not evident in the visible code. The risk classifications (DESTRUCTIVE, READ_ONLY, WRITE) suggest a permission model exists, but it is not exposed in tool definitions where LLMs need it. Overall, this is a D-grade server: major gaps in schema visibility, minimal descriptions, and no discernible parameter or output documentation.
Apply unified diff patches to files
Edit specific portions of a file
Execute system command with input/output handling
Create a followup task for another agent
Retrieve persistent memory or context
Interrupt execution of another agent
List active collaborative agents
List directory contents
List MCP resource templates
No input schemas visible for any of the 23 tools. Per HARD SCORING RULE: tools without visible input schemas score 0 on schema. This is the single largest deficiency.
All tool descriptions are 9 - 37 characters (baseline p10=34, p90=392 chars; LLM-optimal 50 - 200 chars). Per HARD SCORING RULE: descriptions under 20 characters cannot exceed 40 points. Descriptions do not explain WHAT each tool does, WHEN to use it, or WHAT it returns. Examples: 'Shell command execution tool' (28 chars), 'Read file contents' (18 chars), 'Spawn a new collaborative agent instance' (41 chars). LLMs cannot distinguish similar tools or understand prerequisites from these summaries.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 32 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 40 | - | v1 |
List available MCP resources
Read file contents
Read content from an MCP resource
Read skill or capability definition
Search for files matching patterns
Send message to another agent
Shell command execution tool
Shell command tool variant
Spawn a new collaborative agent instance
Update task plan or status
Wait for another agent to complete
Fetch content from web URLs
Write content to a file
Write data to process standard input
No parameter-level descriptions visible for any tool. LLMs cannot infer whether a 'status' param means HTTP status, task status, or active flag. Parameters like 'command' (in shell/exec_command) lack format constraints, character limits, or character restrictions. Parameters accepting file paths (read, write_file, edit_file) lack guidance on whether absolute vs. relative paths are valid, or whether symbolic links are followed.
Destructive tools (shell, shell_command, exec_command, write_stdin, apply_patch, write_file, edit_file, update_plan, spawn_agent, send_message, followup_task, interrupt_agent) lack error handling guidance, confirmation steps, and dry-run capabilities. Per pattern 'confirmation-request': 'Irreversible operations should support a dry-run or confirmation step. Agents make mistakes, a confirm_before_execute pattern prevents catastrophic errors.' No tool description indicates whether a call is retryable, user-fixable, or fatal. The visible approval/gate machinery in sdk_tools.ts and step_gate exist server-side, but are not exposed in tool definitions where LLMs need them.
No output schemas or return value documentation visible. LLMs do not know what fields are returned by list_directory, read_mcp_resource, search_files, or list_agents. Without knowing what data is available, LLMs cannot chain these tools to downstream ones (e.g., knowing that read returns file content as a string, vs. an object with metadata).
Multiple tools with similar names and unclear distinctions: 'shell' vs 'shell_command' vs 'exec_command' (3 variants of shell execution); 'read' vs 'read_skill' vs 'read_mcp_resource' (3 variants of reading). Consolidate or rename for clarity (e.g., 'exec_shell_command', 'read_file', 'read_agent_skill', 'read_mcp_resource').
Pagination and result limits not documented. Tools like 'list_directory', 'search_files', 'list_mcp_resources', and 'list_agents' return lists but lack visible descriptions of pagination, max result count, or pagination parameters (limit, offset, cursor). Without pagination, large results blow the context window.'
Multi-agent tools (spawn_agent, send_message, followup_task, wait_agent, interrupt_agent) lack clear parameter documentation on agent_id resolution (name vs opaque ID), message format constraints, and task structure.
No tool schema reveals parameter types (string, number, boolean, enum, array) or ranges. Per HARD SCORING RULE: 'If parameters lack type definitions, schema score cannot exceed 30.' Since no schemas are visible, all tools score 0 on schema. Critical parameters are undocumented: does 'limit' accept 1 - 100 or unbounded? Does 'pattern' accept regex or glob syntax? Is 'recursive' a boolean or string? Without type information, LLMs guess and send invalid inputs.
File I/O tools (read, write_file, edit_file, apply_patch) do not explicitly document path traversal or symlink handling. Sanitize against SQL injection, command injection, and path traversal.' An LLM tricked via prompt injection could pass '../../../etc/passwd' to read or write_file, with no visible safeguard documented in the tool definitions.