Loading ToolBench
Local-first agent memory: an Obsidian markdown vault as source of truth, with a rebuildable DuckDB index.
agentcairn has 5 tools with moderate definition quality. Tool names are clear and verb-based (search, recall, build_context, recent, remember). Descriptions are present for all tools and reasonably detailed (ranging 60-180 chars), explaining WHAT the tool does and WHEN to use it. Input schemas are visible and properly typed with JSON Schema. However, there are gaps: output schemas are not explicitly documented in the source (only inferred from return type comments), some parameters lack constraints (e.g., 'k' is unbounded integer), and error handling guidance is minimal. The 'remember' tool (WRITE operation) lacks explicit confirmation or dry-run pattern. Tool descriptions are functional but could be more LLM-optimized per the 50-200 char baseline. Two tools (search, recall) have nearly identical descriptions, risking LLM confusion. Overall, this is solid mid-tier work, clear definitions, proper schemas, but missing output documentation and composition patterns.
Return a note plus its 1-hop linked neighbors.
Search then hydrate the top-k notes' full text. Reranks by default (set CAIRN_RERANK=0 to disable, or pass rerank=false). Recall prefers your current project's memories (boosted, non-lossy): pass `project` (a repo name) to target another project, else the server's working directory is used. `scope="project"` hard-limits results to that project.
List the most-recently-modified notes.
Persist a distilled memory (redacted, non-lossy) into the vault.
Hybrid search over memory; returns a compact id+snippet index. Reranks by default (set CAIRN_RERANK=0 to disable, or pass rerank=false). Recall prefers your current project's memories (boosted, non-lossy): pass `project` (a repo name) to target another project, else the server's working directory is used. `scope="project"` hard-limits results to that project.
Output schemas not documented in source code. Tools return JSON but structure/fields are not explicitly declared in descriptions or inline docs. LLMs cannot plan downstream calls without knowing response fields.
search and recall have nearly identical descriptions ('Hybrid search over memory; returns...', 'Search then hydrate...'). LLMs may conflate them or pick wrong tool. Descriptions should clarify the key difference: search returns compact id+snippet, recall returns full text.
Parameter 'k' (number of results) is unbounded integer with no min/max constraints. Description lacks range guidance. LLMs could pass absurd values (k=999999) causing performance issues or timeouts.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | 2026-07-28+ | v2 |
build_context description is extremely terse ('Return a note plus its 1-hop linked neighbors.', 21 chars). Insufficient context for LLM to determine WHEN to call it vs search/recall. Should explain use case: 'Get a note with its directly linked neighbors for context-aware composition.'
recent description lacks clarity on ordering/filtering. 'List the most-recently-modified notes' doesn't explain: by what metric (modified vs created)? From which projects? Sorted ascending or descending? Should add: 'Returns up to n most recently modified notes across all projects, newest first.'
remember (WRITE tool) lacks confirmation or dry-run pattern. Tool modifies state (persists memory) but has no warning, validation feedback, or confirmation step. Agents could accidentally spam duplicate memories without safeguard.
Parameter descriptions use generic language without format constraints. E.g., 'query' is 'The search query text' (no length limits, special char guidance). 'project' is 'Target project name (repo name) or null' but doesn't clarify: which repo names are valid? How to discover them? Should reference discovery tools or constraint format.
No error handling guidance in tool descriptions. If search returns zero results, if remember fails due to vault lock, if build_context finds a note with broken links, LLM gets no recovery hint. Should include: 'If query returns empty, try broader terms or search_recent() for discovery.'