Loading ToolBench
The ACI MCP server has significant structural issues that severely limit tool quality. While the code demonstrates proper MCP registration using fastmcp, the tool definitions suffer from: (1) Dynamically generated tool names that obscure actual tool identity, tools are named via f-string templates like '{group_name}_get' and '{tool_base}_post' rather than concrete names, making it impossible to verify what tools actually exist; (2) Descriptions that are template-based and generic, providing minimal guidance to LLMs on when to invoke specific tools; (3) Input schemas that lack critical detail, parameters like 'query_params' and 'filter_expression' are documented as optional objects/strings with no constraints or format guidance; (4) No output schema documentation whatsoever, tools return raw API responses (dict) with no indication of what fields are present or how to chain calls; (5) Critical missing parameter descriptions for most parameters across all tools; (6) No error handling guidance, when aci_controller.get() or post() fails with a ToolError, the message is bare ('{method} request failed: {e}') providing no recovery path to the LLM; (7) Four tools total, but only one unique action pattern (grouped GET vs ungrouped GET/POST/DELETE), suggesting undersupported CRUD model. The server is architecturally sound (proper use of Pydantic models, token-based auth, timeout handling) but fails to meet agent-quality definition standards.
GET any endpoint from group '{group}' ({count} endpoints). Generated dynamically from urls.json groups.
DELETE resource at {name} in ACI (ungrouped).
GET data for {name} from ACI (ungrouped).
POST (create) data to {name} in ACI (ungrouped).
Tool names are template placeholders, not concrete identifiers. Tools are registered as '{group_name}_get', '{tool_base}_get', '{tool_base}_post', '{tool_base}_delete', the actual names depend on runtime URL parsing. This violates the principle that tool names should be deterministic and self-documenting. LLMs cannot select tools by name when names are unknown until runtime.
Descriptions are generic templates with minimal context. E.g., 'GET any endpoint from group \'X\' (N endpoints)' tells the LLM nothing about what data is returned, when to use it vs other tools, or what prerequisites exist. No description exceeds ~60 characters; baseline for A+ tools is 194 chars. Descriptions provide no WHEN guidance.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 11 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 20 | - | v1 |
Parameter 'query_params' across all tools is documented as 'Optional query parameters' with type 'object' and NO further guidance. LLMs cannot determine what keys are valid, what values are expected, or format constraints.
Parameter 'filter_expression' in grouped GET tool is documented as 'Optional filter expression for the query' with no indication of syntax, format, or ACI-specific constraints (e.g., is this LDAP, XPATH, custom DSL?). LLMs cannot construct valid filters without format guidance.
DELETE tool has ZERO input parameters but no description explaining what is being deleted, how to identify the target, or what ID the tool expects. The function signature is 'def tool() -> dict:' with no inputs, making it impossible for an LLM to invoke safely. Is it deleting the entire ACI fabric? A single endpoint? This is dangerously ambiguous.
No output schema documentation for any tool. Tools return raw dicts from aci_controller methods, but there is no description of what fields are present, their types, or how to chain results into downstream tool calls. E.g., if a GET returns an ID, POST needs to reference it, but response structure is undocumented.
Error handling is minimal. When aci_controller._request() fails, it raises ToolError(f'{method} request failed: {e}'). This gives the LLM no guidance on retry, user-fixable issues, or recovery. No error classification (retryable vs fatal), no suggestion of alternative tools (e.g., 'endpoint not found, try list_endpoints()'), no actionable next steps.
DELETE tool lacks confirmation or dry-run capability. A destructive operation with no safety guardrails invites accidental data loss. Pattern requires confirmation for irreversible operations, but this tool can be called immediately without any prompt.
No pagination support. Tools accept 'query_params' as a catch-all, but there is no documented support for limit/offset or page parameters, no total count returned, and no guidance on handling large result sets. An endpoint returning 10,000 results would blow the context window with no way to subset.
Parameter 'payload' in POST tool is documented as 'JSON payload for POST' with no schema or structure guidance. LLMs must infer valid ACI payload structure from context or trial-and-error, leading to malformed requests.