Loading ToolBench
Leteo defines 3 tools with schemas and descriptions, but quality is inconsistent. mem_save has a complex, poorly-scoped description (200+ chars, mentions internal implementation details like 'candidates' and 'mem_judge'). All three tools lack clear action verbs in naming (mem_* prefix is domain-specific but not action-oriented). Parameter descriptions exist but are sparse (5-15 chars each). No output schemas documented. Error handling guidance is absent. Tools appear to conflate multiple concerns (mem_save handles project resolution, session management, and observation capture in one call).
List observations due for review or mark one reviewed. Actions: list, mark_reviewed. Bodies come back as a 400-character preview marked `content_truncated`; read one in full with mem_get_observation.
Save an observation to persistent memory — a decision, a fix, a discovery, a convention — as it happens rather than at the end. Without session_id, uses a stable manual-save session for the detected project. A reply carrying `candidates` means this may contradict memories already held: settle each one with mem_judge in the same turn, because a pair left unjudged is dropped rather than deferred.
Revise a stored memory. Fields left out keep their current value. The memory comes back as a 400-character preview marked `content_truncated`; read one in full with mem_get_observation.
Tool names lack action verbs. 'mem_save', 'mem_update', 'mem_review' are domain-prefixed but do not start with clear verbs (save_, update_, list_). LLMs infer intent from verb-noun patterns; these names require reading descriptions to disambiguate.
mem_save description conflates multiple concerns: persistent memory, session management, project resolution, and conflict handling ('candidates', 'mem_judge'). This violates single-responsibility principle and confuses LLMs about when to call it.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields (e.g., observation IDs for mem_update). Responses are opaque.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 45 | <=2025-11-25 | v2 |
Parameter descriptions are minimal (5-15 chars). 'Optional session ID; uses stable manual-save session if not provided' is clear, but most params like 'scope', 'kind', 'topic_key' lack context. LLMs cannot infer valid values or constraints.
mem_review action parameter accepts free-form strings ('list', 'mark_reviewed') without enum constraint. LLMs may hallucinate invalid actions. Should declare as enum: ['list', 'mark_reviewed'].