Loading ToolBench
A multi-tool AI assistant server that provides indexing, retrieval, chat, and GitHub bot capabilities with support for multiple AI assistant frameworks (Amp, Claude, CommandCode, Cursor)
This server exposes 5 tools with mixed quality. Three tools (index_repository, index_repository_browser, retrieve) lack visible input schemas in the source code provided, only parameter names are inferred. Two tools (chat, github_webhooks) have explicit schemas with type definitions and constraints. However, all tool descriptions are present but vary in quality. The server lacks comprehensive error handling guidance, parameter validation documentation, and output schema definitions. Tool composition is reasonable but could be improved by unifying index operations and providing clearer chainability. No tool annotations (readOnlyHint/destructiveHint/idempotentHint) are visible.
Post a message to the chat API for context-aware responses using retrieval-augmented generation
Webhook endpoint for GitHub issue_comment events that triggers bot commands (status, cancel, help, and custom configured commands)
Index a repository by walking the file tree, chunking files by type (markdown with semantic sections, others with sliding window), and generating embeddings
Index a repository using Hugging Face transformers for browser-compatible embeddings without requiring API calls
Retrieve relevant context chunks from the indexed repository based on semantic similarity to a query
Three tools lack visible input schemas: retrieve, index_repository, index_repository_browser. Only parameter names are inferred from documentation; no JSON Schema type definitions, constraints, or descriptions are visible in the source code provided.
Tool descriptions lack WHEN/WHY guidance. 'Retrieve relevant context chunks' and 'Index a repository' state WHAT but not WHEN to call them or how they relate to other tools. Missing: prerequisites, typical use sequencing, and disambiguation from similar operations.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields (e.g., does retrieve return chunk_id? path? embedding_distance?). Without documented outputs, agents cannot reliably chain tools.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 52 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 0 | - | v1 |
No error handling guidance. No clear messages for missing indices, invalid queries, malformed GitHub payloads, or indexing failures. Errors appear to return generic HTTP status codes without actionable next steps for the LLM.
Tool composition issue: index_repository and index_repository_browser are nearly identical (same input, both index repos). LLM cannot easily distinguish when to use OpenAI embeddings vs Hugging Face. Missing description of trade-offs (speed, accuracy, offline capability).
github_webhooks tool name does not follow verb_noun convention. 'github_webhooks' is a noun; better names would be 'handle_github_webhook' or 'process_github_event' to signal that the tool performs an action when invoked.
Parameter 'x-hub-signature-256' and 'x-github-delivery' in github_webhooks are HTTP headers, not semantic parameters. Exposing them as input params mixes transport concerns with tool semantics. These should be validated server-side, not exposed as LLM-visible inputs.
retrieve tool accepts topK with only a default (5) but no constraints documented (min, max). LLMs could request 1000 chunks, wasting tokens. Missing: explicit range constraints and explanation of trade-offs (latency, token cost, accuracy).