Loading ToolBench
Validate OpenAPI specs, inspect and run operations, execute collections, and keep apiary-generated specs in sync.
openrun MCP server has well-structured tool definitions with complete JSON schemas and descriptions for all 9 tools. Naming follows verb_noun convention (validate_spec, list_operations, run_operation). However, descriptions are terse (10-50 chars for most tools), lacking context on WHEN to use each tool or dependencies between them. Parameter descriptions exist but are minimal. Error handling is present in code but not documented in tool descriptions. No output schema documentation visible. Tool composition is sound, each tool has one responsibility, but descriptions don't guide LLM selection or explain recovery paths.
Ask apiary whether the saved OpenAPI document matches the current Go source without writing files.
Generate a curl command for one OpenAPI operation without sending it.
Generate OpenAPI through apiary in memory and return a unified diff against the saved spec without overwriting it.
Run apiary and write the generated OpenAPI document to the configured spec path.
List OpenAPI operations, optionally filtered by method, path, operationId, tag, or summary.
Run an openrun collection and return its structured report.
Send one HTTP request generated from an OpenAPI operation and return the response and assertions.
Descriptions are too terse (10-50 chars). Rubric baseline is 194 chars. LLMs cannot determine WHEN to use each tool or how it differs from similar tools (e.g., template_operation vs curl_operation vs run_operation).
No output schema documentation. Rubric requires documented return types for 100% of A+ tools. Callers cannot plan downstream tool calls without knowing what fields are returned.
Parameter descriptions lack format/constraint details. E.g., 'operation' param accepts 'operationId or METHOD /path selector' but no examples or validation rules shown. Rubric requires 'The project key (2-10 uppercase letters, e.g. PROJ)' level of detail.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 63 | 2025-06-18+ | v2 |
Generate an editable request template for one OpenAPI operation.
Load and validate an OpenAPI spec using openrun's runner-facing validation.
No error recovery guidance in tool descriptions. Rubric pattern:recovery-guide requires 'User not found. Try search_users() with a partial name.' style hints. Current descriptions do not guide LLM on what to do if a call fails.
Tool composition clarity: template_operation, curl_operation, and run_operation are similar. Descriptions do not explain the workflow (template → curl → run) or when to use each. LLM may pick wrong tool.