Loading ToolBench

Reading the current corpus…

chat-mcp MCP server · F · ToolBench

chat-mcp

FastAPI-based MCP server gateway that manages multiple MCP servers, lists their tools, and invokes tools across different MCP server instances

2026-07-28+STREAMABLE_HTTP5 tools

37/100

Definition Quality43

Protocol Readiness48

Supportability18

Spec posture

Inferred effective spec 2026-07-28+·current baseline 2026-07-28

Static source inference · medium confidence · evidence: Streamable HTTP

Current-spec patterns detected

Summary

This server acts as a gateway/orchestrator for other MCP servers rather than a direct tool provider. Tool definitions are present with schemas and descriptions, but quality is inconsistent. Naming follows verb_noun convention (add-, list-, invoke-) which is good. However, descriptions are generic and lack context about when/why to use each tool. Parameter descriptions exist but are minimal. Critical gaps: no error recovery guidance, no output schema documentation, no pagination support for list operations, and invoke-mcp-tool accepts arbitrary arguments without validation. The server lacks idempotency guarantees and confirmation patterns for destructive operations (add-mcp-server modifies persistent state). Overall, definitions are functional but fall short of production-grade quality.

Tools (5)

add-mcp-serverwrite50/100

Add a new MCP server to the list of available servers

invoke-mcp-toolwrite50/100

Invoke a tool on a specific MCP server

list-mcp-serversread only50/100

List available MCP servers and their configurations

list-mcp-servers-fullread only47/100

List available MCP servers and their configurations

list-mcp-toolsread only50/100

List tools available from a specific MCP server

Issues (5)

HIGHtool-composition ↗

Duplicate tools (list-mcp-servers vs list-mcp-servers-full) create LLM confusion. Both list servers; only difference is verbosity. Should consolidate into one canonical tool with optional detail parameter.

list-mcp-serverslist-mcp-servers-full

CRITICALconstrained-input ↗

invoke-mcp-tool accepts arbitrary 'arguments' object with no schema, validation, or type hints. LLM cannot know what arguments are valid for a given tool. Requires tool-specific schema introspection or documentation.

invoke-mcp-tool

HIGHresponse-shaper ↗

No output schema documentation. list-mcp-tools returns 'mcp-tools' array but structure is undocumented. invoke-mcp-tool returns result.model_dump() with no schema guidance. LLM cannot plan downstream operations.

list-mcp-toolsinvoke-mcp-tool

HIGHconfirmation-request ↗

add-mcp-server modifies persistent state (mcp_servers.json) with no confirmation, dry-run, or rollback capability. Agents can accidentally register malicious servers. Missing confirmation-request pattern.

Recommendations

Merge list-mcp-servers and list-mcp-servers-full into a single tool with an optional 'include_full_config' boolean parameter (default false). Reduces LLM decision overhead.
Document the output schema for each tool. E.g., list-mcp-tools returns {mcp-tools: [{name: string, description: string, inputSchema: object}]}. invoke-mcp-tool returns {content: [{type: string, text: string}], isError: boolean}.
Add a 'dry_run' parameter to add-mcp-server (default false). When true, validate the server config without persisting. Prevents accidental registration of broken servers.
Enhance error messages with recovery hints. E.g., 'Tool not found. Available tools: [list]. Call list-mcp-tools to discover tools for this server.' This guides the LLM to self-correct.
Add input validation to invoke-mcp-tool. Before calling session.call_tool(), validate that arguments match the tool's inputSchema. Return a clear error if validation fails, e.g., 'Missing required argument: query (string, max 100 chars)'.

Score history

Overall score trend

First recorded score · v2 rubric

37/100

Scored	Grade	Overall	Spec posture	Rubric
2026-09-23	F	37	2026-07-28+	v2