Loading ToolBench
Multi-agent AI orchestrator for IT diagnostics and engineering workflows, exposed as an MCP server to allow any MCP-compatible AI assistant (Claude Desktop, Cursor, Windsurf, etc.) to call orchestrator capabilities as tools.
The server defines 11 tools with explicit names, descriptions, and input schemas. However, quality is uneven: tool descriptions range from adequate (start_run, run_from_template) to bare-minimum (file_read, file_metadata). Most critically, OUTPUT SCHEMAS ARE NOT DOCUMENTED, the rubric baseline requires 100% of A+ tools to document return types, and this server does not. Parameter descriptions are present but many lack constraints (enums, ranges, formats). No error handling guidance is visible in tool definitions. The tool set spans orchestration (start_run, get_run_status, cancel_run, list_templates, list_agent_profiles, list_runs, run_from_template) and file operations (file_read, code_search, directory_list, file_metadata), but file operation descriptions are generic and lack context on when to use them vs. alternatives. No tool annotations (readOnlyHint, destructiveHint, idempotentHint) are present. Security: start_run and run_from_template accept free-form strings (goal, template_name, params) with no validation or injection guards visible in the definitions. Composition: file_read/code_search/directory_list/file_metadata form a coherent file-system toolset, but orchestration tools lack clear chaining guidance (e.g., does get_run_status return the data format that downstream analysis tools expect?). Overall: definitions meet a baseline standard but fall short of production-grade quality.
Cancel a pending or running orchestrator run.
Search codebase for patterns (grep-like functionality). Tool for searching codebase with grep-like functionality and regex support.
List files and directories in a path. Tool for listing directory contents with depth control.
Get metadata about a file (size, modified time, permissions, type). Tool for retrieving file metadata safely.
Read contents of a file (read-only, size-limited). Tool for reading files safely with restrictions on file type, size, and workspace containment.
Get the current status and result of a run.
OUTPUT SCHEMAS NOT DOCUMENTED for any tool. The rubric baseline requires '100% of A+ tools have documented return types.' This server has zero documented response schemas, violating a critical quality requirement. LLMs cannot infer what fields to expect from a response, forcing them to guess at downstream tool chaining.
File operation tools (file_read, code_search, directory_list, file_metadata) have vague, under-100-character descriptions lacking operational detail. LLMs cannot determine when to use code_search vs directory_list vs file_read.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 64 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 40 | - | v1 |
List available agent profiles (specialist configurations for different task types). Use the profile id as agent_profile_id in start_run.
List recent orchestrator runs, optionally filtered by status.
List all available run templates with their parameter schemas. Returns a list of templates, each with: id (the template_name to pass to run_from_template), name (human-readable name), description (what it does), params (required and optional parameters with descriptions).
Start a run from a pre-built template with parameter substitution. Templates are pre-tested IT operations recipes. Call list_templates() first to discover available templates and their required parameters.
Start a new orchestrator run for the given goal. The orchestrator plans and executes multi-step IT operations using its configured MCP tools and LLM.
No result limits or pagination documented for list_* and search tools. An LLM querying a large codebase could fetch 10,000+ results, exhausting context.
Parameters accept free-form strings without validation or injection guards. start_run accepts a 'goal' parameter (any string) and run_from_template accepts 'template_name' (any string) with no enum constraints. This also creates injection risk if these strings are passed to external systems.
run_from_template 'params' parameter is a generic 'object' with no schema. LLMs cannot determine what keys/values are valid; agents will pass incorrect params or miss required ones. The description references list_templates() to discover params, but there is no formal validation schema.
No error handling or recovery guidance in any tool description. If cancel_run tries to cancel a completed run? If file_read encounters a permission error? Without recovery guidance, LLMs cannot self-correct on failure.
cancel_run lacks state-modification warning and destructive operation guidance. No confirmation mechanism is visible. LLMs may cancel runs unintentionally.
File path operations lack explicit security constraints in descriptions. Sanitize against... path traversal.' file_read says 'must be within workspace root' but does not explain: How is workspace root defined? Are symlinks allowed? Are relative paths like '../' rejected? These constraints must be explicit for LLMs to understand safe usage.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) are present. Per current MCP spec (2026-07-28), tool annotations are a recommended pattern for LLM guidance. This server does not use them, forcing LLMs to infer operation safety from descriptions alone, which is error-prone.
Default parameter values may cause unintended behavior. code_search defaults directory='.' and file_pattern='*.py'.