Loading ToolBench
A server to manage OpenShift clusters using the OCM API.
OpenShiftClusterManagerServer demonstrates moderate definition quality with structured tool definitions and reasonable parameter schemas. Both tools have valid input schemas and descriptions, but there are gaps in error handling guidance, parameter constraints, and output documentation. The naming is clear and action-oriented (create_*, get_*), and descriptions are present but lack specificity about failure modes and next steps. Resources are well-defined with URIs and descriptions, but the tools themselves lack output schema documentation and error recovery guidance.
Create a new OpenShift cluster (e.g., ROSA or OSD).
Get the status of an OpenShift cluster.
Output schemas not documented. Neither tool specifies what fields are returned to the LLM. This forces the agent to guess structure of responses and breaks chaining if downstream tools need IDs or status codes.
Error handling lacks recovery guidance. The _ocm_request helper raises generic ToolError with API messages, but does not categorize errors as retryable, user-fixable, or fatal. LLM receives raw HTTP 4xx/5xx codes with no hint about next action.
get_ocm_cluster_status description is vague. 'Get the status of an OpenShift cluster' does not explain what 'status' includes, what fields are returned, or when to use this vs list_all_clusters. Lacks context for tool selection per pattern:tool-description.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 59 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 54 | - | v1 |
create_ocm_cluster has 'properties' parameter (dict[str, Any]) with no constraint or schema. This is a catch-all that invites hallucination. LLM cannot know valid provider-specific properties without enumeration or examples.
compute_machine_type parameter lacks machine type enumeration or validation rules. Description says 'e.g. m5.2xlarge for AWS' but does not specify which machine types are valid per cloud provider. LLM may pass invalid values.
version parameter description contains example value 'openshift-v4.14.0' in text. Per pattern:tool-description, examples in descriptions tempt LLMs to use them literally. Should replace with enum, format constraint, or discovery tool reference (list_ocm_versions).
No confirmation or dry-run pattern for create_ocm_cluster. This is a destructive write operation (creates real infrastructure) but tool executes immediately. Should support confirm_before_execute or dry_run mode per pattern:confirmation-request.
Resources are well-named but lack composition guidance. If an agent calls list_all_clusters then create_ocm_cluster, it needs the cluster_id from list response to use in other tools. Response structure for resources is not documented.