Loading ToolBench
Semantic search over indexed Mattermost history and Claude Code sessions using sentence-transformers embeddings and Chroma vector database
Single tool 'rag_search' has a clear, descriptive docstring (194 chars) and well-defined parameters with types and descriptions. However, the output is returned as a JSON string rather than a structured object, forcing the LLM to parse unstructured text. Parameter descriptions are good but lack explicit constraints (e.g., top_k range, ISO date format validation). No error handling guidance or recovery paths documented. Tool naming is verb-noun compliant ('rag_search' = search action), but the schema lacks formal output documentation.
Semantic search over indexed Mattermost history (and optionally Claude Code sessions).
Output returned as JSON string instead of structured object. LLM must parse unstructured text; no documented output schema. Violates pattern:response-shaper.
Parameter 'top_k' lacks explicit range constraints (min/max). Unbounded integers risk API abuse or timeout. Should document: 'integer, 1 - 100, default 8'.
Parameter 'since' accepts ISO date string but lacks format validation or error guidance. No documented behavior for malformed dates (e.g., '2025-13-45').
No error handling or recovery guidance. Tool may fail silently if Chroma collection is missing, model download fails, or query is malformed. LLM has no guidance on what to do next.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 59 | 2026-07-28+ | v2 |
Tool truncates messages to 800 chars without documenting this behavior. LLM may not realize results are incomplete and could misinterpret truncated context.