Loading ToolBench
Head-to-head benchmark for MCP memory systems — multi-session coding agent scenarios
This benchmark tool exposes 16 memory system adapters with inconsistent quality. Tool names follow verb_noun convention (memory_store, memstate_get, delete_memory), which is good. However, descriptions are generic and lack LLM-optimized guidance on WHEN to use each tool vs. alternatives. Parameter schemas are present but minimal, most lack constraints, enums, or format specifications. No output schemas documented. Error handling is absent; tools return success/failure booleans with no recovery guidance. The mock adapter (tools 1-4) and preset adapters (tools 5-16) duplicate functionality across three memory systems (mock, Memstate, Mem0), creating confusion about which to call. Destructive operations (delete_memory, delete_all_memories, memstate_delete_project) lack confirmation patterns or dry-run support.
Store a fact in Mem0 memory system
Delete all memories for a user in Mem0
Delete a fact from Mem0 memory
Retrieve facts from Mem0 memory system
View the version history of a specific memory in Mem0
Retrieve a fact or browse memory. Leave key empty to browse everything.
View the version history of a specific keypath.
Destructive operations lack confirmation or dry-run support. Tools memstate_delete, memstate_delete_project, delete_memory, and delete_all_memories can permanently erase data with no recovery guidance or user confirmation step.
No output schemas documented. Tools return success/failure booleans with optional 'data' and 'rawResponse' fields, but LLMs cannot plan downstream calls or extract structured results. Baseline: 100% of A+ tools document return types.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 49 | 2026-07-28+ | v2 |
Search memory by meaning using natural language.
Store a fact in the project's memory. Use dot-separated keypaths like 'database.schema.users'. The project name is added automatically.
Delete a fact from Memstate memory
Delete a project's memory in Memstate
Retrieve a fact from Memstate memory system
View the version history of a specific keypath in Memstate
Search Memstate memory by meaning using natural language
Store a fact in Memstate memory system
Search Mem0 memory by meaning using natural language
Descriptions lack LLM-optimized guidance on WHEN to use each tool. 'Store a fact in Memstate memory system' does not explain when to call memstate_set vs. memory_store vs. add_memory. Baseline: descriptions should be 50-200 chars and answer WHAT, WHEN, and WHAT IT RETURNS.
Parameter schemas lack constraints, enums, or format specifications. 'key' and 'keypath' parameters accept free-form strings with no validation rules, format hints, or examples. LLMs cannot infer valid formats (e.g., 'database.schema.users' vs 'database/schema/users').
No error handling or recovery guidance. Tools return success/failure with optional error strings, but do not categorize errors as retryable, user-fixable, or fatal. LLMs receive no guidance on next steps after failure.