Loading ToolBench
A FastAPI-based MCP server management platform that allows users to create, manage, and run MCP servers with tools, resources, and prompts. Supports user authentication, library management, and MCP server inference.
CraftMCP exhibits significant definition quality gaps across 33 tools. While all tools have names and descriptions, most descriptions are generic (avg ~80 chars, well below the 194-char baseline for A+ tools). Input schemas are present but lack depth: parameters have types but many lack descriptions or constraints. No output schemas are documented. Error handling is minimal, most tools return generic HTTPException without recovery guidance. The server implements a meta-tool pattern (create/modify/delete/link/unlink for tools, prompts, resources, MCPs) which is compositionally sound, but individual tool definitions lack the LLM-optimization needed for reliable agent selection and parameter binding.
Create a new MCP server with name, description, custom imports, and global variables
Create a new MCP prompt with name, snippet, optional MCP link, and parameters
Create a new MCP resource with name, path template, snippet, optional MCP link, and parameters
Create a new MCP tool with name, snippet, async flag, optional MCP link, and parameters
Uninstall a Python library via pip and remove it from tracking
Delete an MCP server permanently
Delete a prompt permanently
Generic, short descriptions across all 33 tools (avg ~80 chars vs 194-char baseline). Descriptions like 'List all MCP tools' lack context for when/why to call the tool, forcing LLMs to guess intent.
No output schemas documented for any tool. Agents cannot predict response structure, forcing them to parse unstructured results or make assumptions about field names and types.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 47 | 2026-07-28+ | v2 |
Delete a resource permanently
Delete a tool permanently
Export a complete MCP server with all linked tools, resources, and prompts as a single Python file
Export an MCP server as metadata.json and mcp.py files
Export a prompt as metadata.json and prompt.py files
Export a resource as metadata.json and resource.py files
Export a tool as metadata.json and tool.py files
List or invoke tools, prompts, and resources from a running MCP server via stdio
Install a Python library via pip and track it in the database
Link a prompt to an MCP server
Link a resource to an MCP server
Link an existing tool to an MCP server
List all installed libraries for the authenticated user (or all if admin)
List all MCP servers owned by the authenticated user (or all if admin)
List all prompts owned by the authenticated user (or all if admin)
List all resources owned by the authenticated user (or all if admin)
List all MCP tools owned by the authenticated user (or all tools if admin)
Get the status of all MCP servers owned by the authenticated user (or all if admin)
Modify an existing MCP server's name, description, imports, or globals
Modify an existing prompt's name or snippet
Modify an existing resource's name, path template, or snippet
Modify an existing tool's name, snippet, or async flag
Build, initialize, and run an MCP server with uv venv and install user libraries
Unlink a prompt from one or all MCP servers
Unlink a resource from one or all MCP servers
Unlink a tool from one or all MCP servers
Parameter descriptions missing or minimal. E.g., 'params' in create-tool is described as 'Function parameters as a dictionary' but lacks format/structure guidance. LLMs cannot infer valid parameter shapes.
Error handling returns generic HTTPException without recovery guidance. E.g., 'Unauthorized' or 'Tool not found' gives agents no actionable next step (e.g., 'Try list-tools() to find valid IDs').
Destructive tools (delete-tool, delete-mcp, delete-prompt, delete-resource, delete-library) lack confirmation or dry-run support. Agents can permanently delete resources without safeguards.