Loading ToolBench
Retrieval, memory, context, and quality tools for any LLM workflow. Provides BM25 keyword search, semantic memory management, context compilation with token budgeting, team collaboration features, and diagnostics.
Koshi demonstrates solid tool design with comprehensive descriptions and well-structured schemas. All 6 tools have detailed, LLM-optimized descriptions (150-400 chars) that explain WHEN to call, WHAT it does, and WHAT to give it. Input schemas are complete with types and descriptions for all parameters. However, output schemas are not documented, LLMs cannot see what fields to expect from responses. Error handling guidance is minimal. Tool naming is clear and verb-driven (compile, token_count, version, health, remember, index). The 'format' parameter pattern across tools (supporting 'text' vs 'json' output modes) is well-designed for structured output. No security issues detected (no secrets in params). Composition is sound, each tool has one clear responsibility. Main gaps: missing output schema documentation, no error recovery guidance, and no tool annotations (readOnlyHint/destructiveHint).
WHEN TO CALL: Once you have collected retrieval results AND/OR memories AND need to pack them into a token-budgeted prompt window before sending to an LLM. Call AFTER koshi_search and/or koshi_recall — never in place of them. WHAT IT DOES: Positions retrieved chunks, memories, team context, and the user query inside tokenBudget using a cache-first strategy by default. Deterministic. No LLM, no network. WHAT YOU GIVE IT: systemPrompt + userQuery (required); retrievedContent (sections joined with '\n---\n'); memories (JSON array, or raw koshi_recall output — auto-parsed); teamContext; tokenBudget (default 8192); strategy (CacheOptimized|PrimacyRecency|RelevanceDescending|Chronological). Pass format="json" for a parseable envelope (#66).
WHEN TO CALL: When something is not working as expected — search returns nothing, memories do not survive restart, the team dashboard shows zeros, or the user reports persistence issues. Always preferable to guessing whether persistence is wired correctly. WHAT IT DOES: Reports indexed corpus size, memory store backend + persistence path, team registry state, snapshot-load status, and uptime. Read-only; never mutates state. Pass format="json" for a parseable envelope (#66).
WHEN TO CALL: When you have in-memory documents (parsed JSON, fetched URLs, generated text) to make searchable. For files already on disk, use koshi_index_directory instead. WHAT IT DOES: Chunks and BM25-indexes the documents under a corpus name. Replaces any prior corpus of the same name. Only the default corpus is persisted via KOSHI_INDEX_FILE. WHAT YOU GIVE IT: documents (JSON array of {content, source, type}); optional maxTokens / overlapTokens / corpus name. Pass format="json" for a parseable envelope (#66).
Output schemas not documented. LLMs cannot see what fields to expect from tool responses (e.g., what does koshi_compile_context return? What fields does koshi_health expose?). This forces LLMs to guess at response structure and risks parsing errors.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). koshi_remember is marked WRITE but lacks destructiveHint annotation. koshi_index is marked WRITE but lacks idempotentHint annotation. Agents cannot determine retry safety or side-effect scope without these hints.
Error handling guidance missing. No descriptions of what errors each tool can return, what they mean, or how to recover. E.g., koshi_index could fail if documents are malformed JSON, but no guidance on retry strategy or user-fixable vs fatal errors.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | B | 71 | 2026-07-28+ | v2 |
WHEN TO CALL: When the user states a fact, decision, pattern, or preference that should outlive this conversation (e.g., 'we use PostgreSQL 16', 'decided to skip caching for now', 'always validate inputs at API boundaries'). Do NOT call for ephemeral chitchat or for things the user is asking you (only for things they are telling you). WHAT IT DOES: Persists a structured memory with type/subject/scope. Memories survive restarts by default (saved to <project-root>/.koshi/memory.json; override via KOSHI_MEMORY_FILE or KOSHI_MEMORY_VAULT). WHAT YOU GIVE IT: content + subject (required); type (Fact|Decision|Pattern|Preference); confidence 0-1; source; scope filters (userId/workspaceId/threadId — '*' or empty means globally visible). Pass format="json" for a parseable envelope (#66).
WHEN TO CALL: To check whether a string fits a budget, or to size a prospective chunk before sending it to an LLM. Useful inside loops that build prompts. WHAT IT DOES: Returns token count using the GPT-4 tokenizer plus character count and chars/token ratio. Deterministic. No network. Pass format="json" for a parseable envelope (#66). WHAT YOU GIVE IT: text (the string to measure).
WHEN TO CALL: When the user asks what version of Koshi is running, or before reporting a bug. WHAT IT DOES: Returns the Koshi MCP server version, .NET runtime, OS, process id, and start time. Pass format="json" for a parseable envelope (#66).
Parameter 'format' is repeated across all 6 tools with identical description. This is a cross-cutting concern that could be documented once in a shared pattern or capability declaration rather than duplicated in every tool.
koshi_index 'documents' parameter expects JSON array but description does not specify the exact schema (e.g., required fields, field types). LLMs may pass malformed documents. Should specify: [{"content": string, "source": string, "type": string}] with field descriptions.