Loading ToolBench
A FastAPI-based server that organizes code files into structured projects and integrates AI provider endpoints (OpenAI, Google) for text generation with project versioning and API key management.
CodePortal MCP exhibits pervasive quality gaps across naming, descriptions, schemas, and error handling. While 24 tools are registered, most lack proper MCP-style tool isolation and parameter rigor. Many tools expose internal FastAPI implementation details (app objects, callbacks) as parameters, violating secret injection and abstraction principles. Descriptions are present but generic (avg ~80 chars, well below the 194-char baseline for A-tier tools). Parameter descriptions are sparse or missing entirely. Tools like 'start_inactivity_monitor' and 'register_ai_endpoints' accept non-serializable objects (FastAPI app instances) as parameters, which is fundamentally incompatible with MCP's stateless, language-agnostic design. No tool documents output schemas. Error handling is absent, no recovery guidance, no invalid-value feedback, no classification of retryability. Security is critically compromised: ai_providers.py functions suggest API keys may be stored in plaintext and passed around without vault-based injection. Overall, the server reads as a FastAPI app with loose tool-like wrappers rather than a properly designed MCP server.
Simple web UI for interacting with AI providers
Create a desktop shortcut for CodePortal with custom icon
Create a new project with files and structure (implied from ProjectRequest model)
Generate text using the specified AI provider
Validate the API key for request authentication
Get API key for a specific provider from archive storage
Get all API keys from archive storage
Non-serializable parameters in tool schemas (FastAPI app instances, callback functions). Tools 'start_inactivity_monitor' and 'register_ai_endpoints' accept 'app' as an object type and 'shutdown_callback' as a string function reference. MCP requires all parameters to be JSON-serializable primitives. These violate the protocol's language-agnostic design.
API keys and secrets passed/stored without vault injection. Code references ai_providers.load_api_keys(), update_api_key(), and get_api_key() functions that likely load credentials from plaintext config.json or environment without proper secret-server abstraction. Tools expose key management directly to clients without encryption or server-side-only storage. Credentials must never appear in tool parameters or responses.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 54 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 33 | - | v1 |
Get the most recent projects from history
Send a request to Google's Generative AI API
Welcome page with basic server info and status
Create necessary directories for AI features
Install required Python dependencies via pip
List available AI provider keys (without showing the actual keys)
Load configuration from config.json file
Load the script_starter.py file and initialize project manager
Log a project into the versioning history file
Send a request to OpenAI chat completion API
Print instructions for a project to a timestamped file
Register AI provider endpoints with the main FastAPI app
Save configuration to config.json file
Start monitoring for user inactivity and shutdown server if inactive
Update the last activity timestamp
Update API key for a specific AI provider in archive storage
Update API key for a specific provider
No output schemas documented. The codebase provides input parameter schemas via Pydantic models (AIRequest, KeyUpdateRequest), but no tool explicitly documents what fields it returns. LLMs cannot plan downstream tool calls or extract structured data from responses. Example: generate_text returns a dict with 'error', 'text', 'model', but this is inferred from code, not declared in tool metadata.
Descriptions are generic and lack LLM-optimization guidance. Average description length ~80 chars (below 194-char baseline). Many descriptions do not explain WHEN to use the tool or what distinguishes it from similar tools. Example: 'get_api_key' description says 'Validate the API key for request authentication', but does this retrieve a key, validate an existing one, or both? Descriptions do not state if operations are side-effecting.
Parameters lack descriptions or have vague descriptions. Example: 'update_key' has 'additional_info' (type: object) with description 'Optional additional configuration fields', what fields? What are their types and constraints? 'additional_config' in 'start_inactivity_monitor' similarly provides no guidance. LLMs cannot fill in object/array parameters without clear field documentation.
No error handling or recovery guidance. Tools do not document failure modes, which errors are retryable, or how to self-correct. Example: generate_text returns {'error': result['error']} on API failures, but does not guide the LLM on retry strategy, rate-limit backoff, or missing authentication. create_project likely fails silently if the directory exists; no guidance on retry vs. collision handling.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). The spec (2026-07-28) includes tool annotations to help agents reason about side effects. 'delete_project' (implied) and 'save_config' are destructive and should be marked; 'list_keys' and 'get_api_key' are read-only. Without annotations, agents may misplan or inadvertently cause data loss.
Overly broad parameter types. 'create_project' accepts 'files' as type 'object' with description 'Dictionary of filename to content mappings', but what are valid keys? String paths only? No nested dirs? What content types (string, binary, unicode)? Without constraints, LLMs guess and pass invalid structures.
Duplicate/overlapping tool definitions. 'get_api_key' (with 'api_key_header' param) vs 'get_api_key' (with 'provider' param) appear to serve different purposes but share the same name. Similarly, 'update_key' (ai_endpoints.py) and 'update_api_key' (ai_providers.py) both update keys but with different signatures. LLMs will conflate these and pick the wrong one.
Tools mixing internal implementation with external interface. 'home' returns 'Welcome page with basic server info and status', but there is no schema; likely an HTML response. 'ai_ui' returns HTML. Web UI endpoints (routes returning HTMLResponse) should not be MCP tools; they conflate HTTP routing with MCP tool design. MCP tools should return structured data, not web pages.
No pagination or result limits documented. Tools like 'get_recent_projects' accept 'limit' parameter (default=3), but no documentation of max limit, whether limit=0 is valid, or how to fetch beyond the limit. 'get_api_keys' returns all keys with no pagination mechanism, if there are thousands of keys, the response balloons the context window.