Loading ToolBench
This server has 11 well-defined tools with complete JSON schemas generated via Zod and proper parameter descriptions. Naming follows verb_noun conventions clearly (mem_it, create_note, get_note, delete_note, list_notes, search_notes, create_collection, get_collection, delete_collection, list_collections, search_collections). Tool descriptions are present and reasonably clear (range 80 - 160 chars). However, there are notable gaps: (1) Output schemas are not explicitly documented in the definitions, tools return JSON.stringify() of unmarked objects, so LLMs cannot predict response structure. (2) Several error conditions lack recovery guidance (e.g., delete_note returns isError but no hint on why deletion failed or what to try next). (3) Tool descriptions do not consistently state WHEN to use a tool vs. alternatives, e.g., search_notes vs. list_notes distinction is not explained. (4) No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite having clear READ_ONLY and DESTRUCTIVE operations. Baselines: tool descriptions average 110 chars (acceptable, target 194); parameters average 4-6 per tool (good); all params have type definitions (✓); most params have descriptions (✓). Pagination is well-implemented with cursor-based limit/page parameters. API secrets are injected server-side (via env.MEM_API_KEY). One tool (mem_it) lacks a clear counterpart (no mem_forget or mem_update), making composition incomplete.
Create a new collection in mem.ai.
Create a new note in mem.ai with markdown content.
Delete a collection from mem.ai.
Delete a note from mem.ai.
Retrieve a specific collection by its ID.
Retrieve a specific note by its ID.
List all collections in mem.ai with optional sorting and cursor-based pagination.
Output schemas not documented. Tool responses return JSON.stringify() of unmarked objects. LLMs cannot predict response structure (e.g., does list_notes return {notes: [...], next_page: '...'} or {data: [...], pagination: {...}}?). This violates pattern:response-shaper and forces LLMs to guess field names for downstream composition.
Destructive tools (delete_note, delete_collection) lack confirmation or dry-run support. No confirmation_request pattern implemented. Agents can irreversibly delete data without verification, increasing operational risk.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite having clearly distinct operation types. READ_ONLY tools (get_note, list_notes, search_notes) are unmarked; DESTRUCTIVE tools (delete_note, delete_collection) are unmarked. This prevents agents from reasoning about side effects and state changes.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 68 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 0 | - | v1 |
List all notes in mem.ai with optional filtering, sorting, and cursor-based pagination.
Remember any content intelligently. This is the primary endpoint for saving information to mem.ai.
Search across all collections in mem.ai with optional query and configuration.
Search across all notes in mem.ai with optional query and advanced filtering options.
Error responses (in server.ts CallTool handler) return isError: true with generic error messages but no recovery guidance. When a tool fails, the error message does not suggest what the agent should try next (e.g., 'User not found, try search_users() first'). Violates pattern:recovery-guide.
Tool descriptions do not explain WHEN to use one tool vs. a similar tool. E.g., search_notes vs. list_notes, both can retrieve notes, but the distinction (query vs. no query) is not stated. This forces LLMs to guess or try both, wasting calls.
No idempotency guarantees documented. If an LLM retries create_note due to a transient network error, will it create a duplicate? Tools should declare idempotency (e.g., via an idempotent_key parameter or annotation) so agents know it's safe to retry.
mem_it tool is a write operation but its description ('Remember any content intelligently') is vague. Does it create a note, add to an existing collection, or annotate a resource? The relationship to create_note is unclear, is mem_it a higher-level abstraction, or do both do the same thing?
Response field naming not validated. If list_notes returns {notes: [...]} but get_note returns {note: {...}}, LLMs must remember two different field names for the same concept. Inconsistent naming breaks composability.