Loading ToolBench
MCP server registry, workspace bindings, and tool catalog for Urule
Gateway server with 13 tools managing MCP server registry, workspace bindings, and tool catalog. Strengths: all tools have descriptions (194-char baseline met), clear verb-noun naming (list_, register_, get_, remove_, search_, create_), and structured input schemas with type definitions. Weaknesses: output schemas are not documented in source code; parameter descriptions are minimal (often 1-2 lines, below 72-char baseline); no error recovery guidance in descriptions; no tool annotations (readOnlyHint, destructiveHint); missing idempotency guarantees for write operations; no pagination metadata (total count, next_cursor) visible in list responses.
Bind an MCP server to a workspace. Creates a binding row that exposes a registered MCP server inside a workspace. Optional `config` is merged at runtime (env vars, per-workspace overrides). 404 SERVER_NOT_FOUND when the referenced `serverId` is not in the registry.
Get MCP server by ID. Returns the registered MCP server record. 404 SERVER_NOT_FOUND when the id is unknown.
Get tool by ID. Returns a single catalog entry. 404 TOOL_NOT_FOUND when the id is unknown.
List registered MCP servers. Paginated list of every MCP server registered with the gateway. `?limit` capped at 100, default 50; `?offset` defaults to 0.
List tools registered for a server. Returns every tool the catalog tracks under this MCP server id. 404 SERVER_NOT_FOUND when the server id is unknown.
List bindings for a workspace. Returns every binding row attached to the given workspace, including per-binding `config` overrides.
Output schemas not documented. No visible return type definitions for any tool. LLMs cannot plan downstream calls or extract required fields (e.g., does list_mcp_servers return pagination metadata? does get_mcp_server return transport type?). Violates pattern:tool requirement that output structure be explicit.
Pagination not fully specified. list_mcp_servers, search_tools, and list_server_tools accept limit/offset but no visible total_count or next_cursor in response. Without pagination metadata, LLMs cannot determine if more results exist or how to fetch them. Violates pattern:paginated-result.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 62 | 2026-07-28+ | v2 |
List MCP servers exposed in a workspace. Returns the server records (resolved via bindings) that are reachable inside the workspace.
List tools exposed in a workspace. Aggregates the tool catalog across every MCP server bound to the workspace, so callers see one flat list of callable tools.
Register a new MCP server. Adds an MCP server record (stdio command, args, optional env, transport hint). Admin-only — the server registry is global. Emits an audit event for the create.
Register the tool catalog for a server. Bulk-registers the tools an MCP server exposes (`name`, `description`, optional `inputSchema`). 404 SERVER_NOT_FOUND when the server id is unknown.
Remove an MCP server. Deletes the server record + every tool the catalog tracked under it. Emits an audit event for the delete. 204 on success, 404 SERVER_NOT_FOUND when unknown.
Remove a workspace binding. Detaches an MCP server from a workspace. 204 on success, 404 BINDING_NOT_FOUND when the binding does not exist.
Search / list the tool catalog. Returns tools across every registered MCP server. `?search=` matches name + description (case-insensitive substring), `?serverId=` filters to a single server. `?limit` capped at 100, default 50; `?offset` defaults to 0.
No tool annotations. Destructive tools (remove_mcp_server, remove_workspace_binding) lack destructiveHint. Read-only tools lack readOnlyHint. Idempotent operations lack idempotentHint. LLMs cannot infer safety/retry semantics without explicit annotations.
Error recovery guidance missing. Descriptions mention 404 codes (e.g., 'SERVER_NOT_FOUND', 'TOOL_NOT_FOUND') but do not guide LLM on recovery. Should say 'Try list_mcp_servers() to find valid IDs' or 'Try search_tools() with a partial name.' Violates pattern:recovery-guide.
Parameter descriptions are minimal. Most params have 1-2 line descriptions (e.g., 'The MCP server ID', 'Maximum number of results'). Baseline is 72 chars; these average ~40 chars. Missing context on format, constraints, and dependencies (e.g., does serverId accept names or only ULIDs? what happens if config overrides conflict?).