Loading ToolBench
Git-based template for private MCP server registries
8 tools with mixed quality. Most have descriptions (good), but parameter documentation is inconsistent. Schemas are present but lack depth, many parameters missing type info or descriptions. Tool naming follows verb_noun convention (fetch_, add_, compile_, validate_) which is solid. However, several tools lack output schema documentation, and error handling guidance is minimal. No tool annotations (readOnlyHint/destructiveHint) despite clear risk profiles. Average description length ~120 chars (within baseline 194), but parameter descriptions are sparse or generic.
Main entry point for add command. Creates private MCP server definitions.
Compile a complete registry from all sources. Process order: 1. Fetch from public registries in order 2. Load private servers 3. Check for conflicts 4. Return merged result
Fetch all servers from the MCP registry using cursor-based pagination.
Fetch servers from a public registry based on config. Handles: servers: '*' (all servers, with optional exclude list), servers: {'name': 'version', ...} (specific servers), servers: {'author/*': 'version', ...} (all servers from author)
Fetch all servers from a registry, handling pagination.
Fetch a specific server version from a registry.
Output schemas not documented. Tools return data but LLMs cannot infer structure (e.g., fetch_all_servers yields servers, but field names/types not specified). Breaks downstream tool chaining.
Parameter descriptions missing or generic. 'registry_config' described as 'Registry configuration dictionary', no hint of structure, required keys, or format. 'env_vars' in add_server lacks type info (array of what?). LLMs cannot validate inputs.
No tool annotations despite clear risk profiles. add_server is WRITE (destructive), but no destructiveHint. fetch_* tools are READ_ONLY but lack readOnlyHint. Agents cannot reason about safety/idempotency.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 52 | 2026-07-28+ | v2 |
Validate all configuration files in the registry.
Validate a server.json file against its declared schema.
Error handling lacks recovery guidance. No indication of what to do on timeout, network failure, or validation error. 'Request timeout in seconds' parameter present but no guidance on retry strategy or fallback.
Pagination not fully specified. fetch_all_servers uses cursor-based pagination (good), but output schema doesn't document cursor field or total count. LLMs cannot determine when to stop or how to resume.