Loading ToolBench
Cross-session memory and recall for AI agents — git-synced knowledge base, knowledge graph, confidence scoring, hybrid semantic+TF-IDF search, auto-distillation with secrets scrubbing
Five tools with comprehensive schemas and detailed descriptions. All tools have input schemas with proper JSON Schema structure, enums, and parameter descriptions. Tool names follow verb_noun convention (knowledge, knowledge_search, knowledge_session, knowledge_admin, knowledge_graph). Descriptions range 150-400 chars, exceeding the 10-1024 baseline. However, output schemas are not documented, LLMs cannot predict response structure. Error handling guidance is absent; tools do not explain recovery paths. Parameter descriptions are thorough but some lack actionable constraints (e.g., token_budget default 800 is stated but not justified). No tool annotations (readOnlyHint, destructiveHint) despite clear risk levels (WRITE vs READ_ONLY marked in metadata but not in schema).
Knowledge base CRUD, sync, and session-start hydration. Actions: "list" (browse entries), "read" (get entry content), "write" (create/update entry, auto git sync), "delete" (remove entry, auto git sync), "sync" (manual git pull + push), "wakeup" (return token-budgeted section-priority context bundle — identity, active_tasks, recent_decisions, known_gotchas, last_session_summary, top_weighted, semantic_fallback — call once at session start).
Admin operations: view vector store stats, view/update configuration, rebuild embeddings, prune orphan session embeddings, or VACUUM the database. Use action "status" for index stats, "config" to view or update settings, "rebuild_embeddings" to re-embed all knowledge entries (useful when switching providers), "prune_orphans" to delete embeddings for sessions no longer present on disk, "vacuum" to reclaim free pages.
Knowledge graph operations: view nodes and relationships, analyze structure, or manually link entries. Use action "nodes" to list all entries, "edges" to list relationships, "analyze" for structural insights (god nodes, bridges, gaps), "link" to create a relationship, "unlink" to remove one.
Search across sessions AND knowledge entries. Returns `{mode, sessions, knowledge}`. General mode (no `scope`): hybrid TF-IDF + semantic over both sources, with optional MMR diversity and category boost. Scoped mode (`scope` set): sessions-only filtered recall for a specific domain (errors, plans, configs, tools, files, decisions).
Output schemas not documented. LLMs cannot predict response structure for any tool. knowledge_search returns {mode, sessions, knowledge} but structure of sessions/knowledge objects is undocumented. knowledge_admin.status returns 'vector store stats' but fields are unspecified.
No error recovery guidance. Tools do not explain what to do on failure. E.g., knowledge.write with git sync could fail due to merge conflicts, but no guidance on retry strategy or fallback. knowledge_search with semantic=true could fail if embeddings unavailable, response mentions fallback but tool description does not.
Tool annotations missing. knowledge (WRITE risk) and knowledge_admin (WRITE risk) lack destructiveHint. knowledge_search and knowledge_session (READ_ONLY) lack readOnlyHint. Metadata marks risk levels but schema does not expose them to LLM.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 62 | 2026-07-28+ | v2 |
Session operations: list sessions, get a full conversation, or get a summary. Use action "list" to browse sessions, "get" to retrieve messages, "summary" for a quick overview.
Parameter constraints lack actionable detail. knowledge.token_budget defaults to 800 (chars/4 estimate) but LLM cannot infer why or what happens if exceeded. knowledge_search.mmr_lambda ranges 0-1 but no guidance on typical values or when to use. knowledge_admin.min_score defaults 0.5 but no explanation of scale or interpretation.
knowledge_admin.promote action has complex multi-parameter logic (min_score, min_recall_count, min_unique_queries) but no example of how gates interact or what 'promotion' means operationally. LLM cannot reason about parameter combinations.