Loading ToolBench
HTTP MCP server for S3 documentation with RAG using HNSWLib and Ollama
Server demonstrates solid fundamentals with three well-defined tools using explicit schema registration via Zod and the MCP SDK. All three tools have action-verb names (search_, refresh_, get_) and substantive descriptions (194-342 chars). Input schemas are properly typed with Zod constraints and descriptions for all parameters. Output schemas are documented via Zod objects. However, several pattern gaps prevent a higher score: (1) output schema descriptions lack depth about field semantics and chaining IDs; (2) parameter constraints are documented in descriptions rather than as formal enum/pattern constraints where appropriate; (3) error handling guidance is minimal, no recovery hints or classification; (4) composition could be stronger, the refresh_index tool's 'force' parameter has a lengthy warning in the description rather than being a separate tool or clearer enum. The server follows good practices around naming clarity (search_documentation vs refresh_index vs get_full_document are distinct) and avoids generic names. Parameter naming is mostly consistent (s3_key is explicit; query is clear). Overall: a B+ implementation with strong scaffolding but missing sophistication in error paths and output richness.
Retrieves the complete content of a Markdown file from S3, along with its metadata (size, last modification date, ETag, chunk count).
Refreshes the documentation index by synchronizing with S3. Automatically detects new files, modifications, and deletions. By default, performs INCREMENTAL sync (fast, only processes changes). Use force parameter ONLY when user explicitly requests a complete rebuild.
Semantic search in documentation stored on S3. Uses local embeddings and a vector store to find the most relevant passages.
Output schema descriptions lack field-level guidance. The search_documentation response returns 'results' (array), 'context' (string), and 'total_results' (number), but descriptions do not explain what 'context' aggregates or whether 'score' is 0 - 1 normalized or raw. This forces LLMs to infer field semantics.
refresh_index 'force' parameter validation is embedded in description text (lengthy warning) rather than as a formal constraint or a separate tool path. The description warns against setting force=true 150+ words, but LLMs may miss this nuance. Consider: (1) enum-style constraint with guidance, or (2) split into two tools: refresh_index (incremental) and rebuild_index (full), making intent explicit.
Error handling does not provide recovery guidance. Tools catch errors and re-throw, but responses lack actionable next steps. E.g., if S3 access fails, tools should suggest: 'S3 credentials invalid, verify AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.' Currently, errors bubble up without context.
Inferred effective spec: 2025-06-18+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-21 | B | 70 | 2025-06-18+ | v2 |
| 2026-03-09 | C | 69 | - | v1 |
search_documentation returns 'max_results' as optional (default: 4) but does not document min/max bounds or the cost of large result sets. If an LLM passes max_results=10000, embedding/vector lookup could timeout or consume excessive memory. Specify: max_results must be 1 - 100 (default 4).
Output from refresh_index includes detailed metrics (documents_scanned, documents_added, documents_modified, documents_deleted, documents_unchanged, errors_count) but no guidance on interpreting anomalies. E.g., if documents_deleted > 0, should the LLM warn the user? No indication. Add advisory field or update description with interpretation hints.
get_full_document parameter 's3_key' accepts a free-form string (e.g., 'docs/authentification_magique_symfony.md') but lacks format validation or examples. Description does not indicate: (1) whether paths are case-sensitive; (2) valid character set; (3) whether leading '/' is required; (4) example valid/invalid keys. Add: 's3_key: string (case-sensitive, alphanumeric + dash/underscore/slash, no leading slash; e.g., "docs/auth.md").'