Loading ToolBench
ERINYS demonstrates solid foundational quality with 20 well-named tools following verb_noun conventions (save_, update_, delete_, search_, get_, list_, create_, traverse_, start_, end_, backfill_, supersede_, query_, distill_, assess_, health_check). All tools have descriptions (avg ~100 chars, within 10-1024 range). Input schemas are present and typed for all tools. However, critical gaps exist: (1) Output schemas are NOT documented anywhere in the source, LLMs cannot predict return structure; (2) Parameter descriptions are minimal (avg ~40 chars) and lack actionable constraints (no enums for obs_type/scope/distillation_level despite fixed valid sets); (3) Error handling is generic (_envelope wrapper) with no recovery guidance; (4) No tool annotations (readOnlyHint/destructiveHint) despite clear risk classifications; (5) Composition issues: backfill_plan and backfill_apply are nearly identical, forcing LLM to reason about mode parameter rather than distinct tools. Strengths: consistent naming, all params typed, metadata handling shows sophistication.
Assess whether retrieved results can answer a given query
Apply a planned backfill operation to import observations
Plan a backfill operation to import observations in batch
Rollback a completed backfill operation
Create a directed relationship between two observations
Delete an observation from memory
Distill an observation to a higher level of abstraction
End the current memory session and optionally save a summary
Output schemas not documented. LLMs cannot predict return structure for any tool. No documentation of fields, types, or pagination for list/search results.
Enum constraints not declared in schema. obs_type, scope, distillation_level, relation, direction have fixed valid sets (defined in VALID_TYPES, VALID_SCOPES, VALID_DISTILLATION_LEVELS, VALID_RELATIONS) but are exposed as free-form strings, inviting hallucinated values.
No tool annotations (readOnlyHint/destructiveHint/idempotentHint). Tools are classified by Risk (READ_ONLY, WRITE, DESTRUCTIVE, REVERSIBLE) in comments but not exposed via MCP annotations, preventing LLM from reasoning about safety without reading descriptions.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 53 | <=2025-11-25 | v2 |
Get statistics about the memory database
Retrieve a single observation by ID
Retrieve recent memory sessions
Check the health and readiness of the ERINYS server
List observations with optional filtering and pagination
Query observations as they existed at a specific point in time
Save a new observation to memory with automatic embedding and collision detection
Search observations using hybrid retrieval (vector + lexical) with RRF ranking
Start a new memory session with optional context
Mark an observation as superseded by a newer version
Traverse the observation graph starting from a given observation
Update an existing observation's content and metadata
backfill_plan and backfill_apply are nearly identical (same parameters, differ only by mode='PLAN' vs 'APPLY'). Should be split into distinct tools or mode should be an enum with clear documentation of state transitions.
Error handling provides no recovery guidance. _envelope() catches exceptions and returns error codes (DUPLICATE, VALIDATION, NOT_FOUND, POLICY_DENIED) but does not suggest next steps (e.g., 'Try search_observations() with a partial title' for NOT_FOUND).