Loading ToolBench
Analytics tool for monitoring AI coding agents across multiple users, hosts, and projects. FastMCP server exposing interaction search (SQL + Qdrant semantic), knowledge base indexing, and sync trigger.
The server provides 8 tools with clear verb-noun naming (get_, semantic_, sync, kb_*) and complete input schemas. Descriptions are present and moderately detailed (avg ~120 chars per tool). However, there are significant gaps: (1) Output schemas are completely undocumented, no return type definitions exist in the source code, (2) API key parameters exposed directly violate secret-injection pattern, (3) Error handling is minimal, no recovery guidance or error categorization, (4) Parameter descriptions lack format/constraint details (e.g., date format YYYY-MM-DD stated but not validated; no enum for 'scope' or 'classification' values), (5) Composition issues, kb_index relies on optional Haiku-based tagging that could silently fail. This is typical mid-tier quality: functional but would not pass principal engineer review.
Get full interaction transcript by session ID.
List interactions with optional filters.
Index an interaction into the semantic knowledge base (Qdrant). Retrieves the interaction from SQLite, generates metadata via Haiku (or uses pre-computed metadata), embeds the content, and stores in Qdrant.
Remove an interaction from the semantic knowledge base.
Check if an interaction is indexed in the knowledge base.
Update tags for an indexed interaction without re-embedding.
No output schemas documented. Tools return results but LLMs cannot plan downstream calls or extract required fields. For example, get_interactions must return a list with fields (session_id, timestamp, project, user, etc.) but this is never defined in the MCP registration.
API keys exposed as tool parameters (api_key, LAV_READ_API_KEY, LAV_API_KEY) violate secret-injection pattern. Keys appear in tool invocation logs and LLM context, risking credential leakage. Must use server-side injection via environment variables.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 61 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 0 | - | v1 |
Semantic search in the indexed knowledge base (Qdrant).
Trigger data sync/re-parse. Requires API key.
Parameter descriptions lack actionable format/constraint details. 'scope' accepts 'all', 'project', 'source' but is described as a free-form string with no enum. 'classification' values (development, meeting, analysis, brainstorm, support, learning) not formally declared as enum. Dates stated as YYYY-MM-DD but no regex pattern or minimum/maximum length constraints documented.
No error handling guidance. Tools can fail (e.g., Qdrant unavailable, session not found, parse error in kb_index) but error responses provide no recovery path. Missing: error categorization (retryable vs. user-fixable vs. fatal), actionable error messages, and remediation suggestions per pattern:recovery-guide.
Destructive operations (kb_remove) lack confirmation step. An LLM can permanently delete an interaction from the knowledge base with a single tool call. No dry-run option, no explicit confirmation requirement. Violates pattern:confirmation-request.
Conditional parameter dependencies undocumented. kb_index has 'pre_metadata' as an optional JSON string, if provided, Haiku auto-tagging is skipped, but this dependency is never stated. 'scope' in sync requires 'project' when scope='project' but describes this only in the project param description, not clearly in both. LLMs may misuse without explicit cross-parameter documentation.
Pagination support incomplete. get_interactions offers 'limit' but no 'offset' or 'next_cursor' for continuation. semantic_search offers 'limit' but no mechanism to retrieve page 2. Large result sets cannot be handled without risking context window overflow. Missing from description: actual max result cap and pagination strategy.
Stateless operation unclear. kb_index with 'full_reparse=true' and 'scope' variations (all/project/source) suggest multi-step or session-dependent behavior. No documentation of whether tool is idempotent or what repeated calls with identical params will do. Pattern:idempotent-operation requires clear guarantees.