Loading ToolBench
Python-based MCP server with orchestration layer, AI engine, and desktop/web UI. Manages tool routing, security filtering, and intelligent pack-based execution.
Matrix MCP v2.0 exhibits severe definition quality gaps across all dimensions. Of 32 tools, most lack proper input schemas (only 4-5 have visible JSON Schema definitions). Descriptions are present but generic and often under 50 chars, failing to explain WHEN to use tools or what they return. Parameter descriptions are sparse or missing entirely. No output schemas documented. Tool names are inconsistent (mix of verb_noun and noun_verb). Critical security tools like 'call_tool_sync' and 'save_api_key' lack error handling guidance. The codebase shows infrastructure/orchestration tools rather than user-facing agent tools, which limits applicability to typical MCP use cases.
Adds engine and project root to Python sys.path for imports.
Synchronously calls an MCP tool by name with arguments. Returns JSON result or raw text.
RouterLock security layer: validates tool calls against whitelist, blocks unauthorized calls, logs violations in JSON format.
Clears AI instance cache.
Deletes profile and associated API key file.
Deletes server configuration file.
Detects user intent from message using IQ-Pack's intent_map.json. Falls back to 'general_chat' if no pack is active.
Missing output schemas: 18+ tools (get_security_report, clear_cache, load, save, get_history_dir, get_profile_dir, get_server_dir, get_prompt_dir, get_settings_file, get_intent_map_file, add_engine_to_sys_path, etc.) have no documented return types. LLMs cannot plan downstream tool calls or extract required fields.
Generic/incomplete descriptions: 15+ tools have descriptions under 50 chars or lack context on WHEN to use them. E.g., 'clear_cache' (25 chars), 'load' (30 chars), 'get_history_dir' (25 chars). Descriptions must explain purpose, prerequisites, and when to select over similar tools.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 41 | <=2025-11-25 | v2 |
Filters tool calls based on intent and whitelist. In universal mode, allows all tools. Logs filtered results.
Generates unique identifier for AI model + version combination (e.g., 'gpt-4-0613').
Gets or creates cached AI instance for specific model + version combination.
Loads API key from profile JSON file (file-based single source of truth).
Returns list of available tools for a given intent. In universal mode, returns empty list (all tools allowed).
Returns path to configuration file.
Returns path to history directory.
Returns path to intent_map.json file for active pack (used for intent detection).
Returns path to profile directory (pack-aware).
Retrieves profile metadata by profile ID.
Returns path to prompt directory (pack-aware).
Returns JSON security report of blocked tool calls and violation count.
Returns path to server configuration directory.
Returns path to settings.json file for active pack.
Returns path to UI file (index_pro.html or fallback to index.html).
Loads configuration from config file and server settings from individual JSON files. Handles legacy migration.
Logs RouterLock security event (allowed/blocked) to audit log with tool name and reason.
Logs tool call event to JSONL audit log with tool name, arguments, result, and error status.
Executes multiple AI functions in parallel with timeout protection. Returns dictionary of results by AI name.
Executes async AI function with retry logic on transient failures. Does not retry on timeout.
Executes async AI function with timeout protection. Returns AIResult with status, data, and execution time.
Saves common configuration to config file (server settings saved separately).
Saves API key to profile JSON file.
Saves server configuration to individual JSON file in server directory.
Sets the active intelligence pack (IQ-Pack) for the session.
No error handling guidance: Destructive tools (delete_server, delete_profile) and security-critical tools (call_tool_sync, save_api_key) lack error recovery instructions. LLMs cannot determine if errors are retryable, user-fixable, or fatal.
Inconsistent naming conventions: Mix of verb_noun (filter_tool_calls, detect_intent) and noun_verb (load, save, clear_cache). Generic verbs like 'load' and 'save' lack specificity. 'load' could mean load config, load cache, or load state, ambiguous for LLM selection.
Secrets exposed in parameters: 'save_api_key' and 'get_api_key' accept API keys as tool parameters. Agent traces log all parameters, secrets in params leak into logs and prompt history. Use server-side secret injection via environment variables or vault.