Loading ToolBench
The server exposes 5 GraphQL query tools with well-structured schemas and clear descriptions. Naming follows verb_noun convention consistently (get-incidents, get-reports, etc.). All tools have explicit input schemas with proper JSON Schema types and descriptions. However, output schemas are not documented, the evaluator cannot verify what fields are returned or their types, forcing downstream agents to discover structure at runtime. Parameter descriptions are present but generic; they lack actionable constraint documentation (e.g., 'limit' accepts integers but no min/max stated). Error handling guidance is absent, tools do not document what failures are retryable or what recovery steps agents should attempt. No tool annotations (readOnlyHint, idempotentHint) are visible in the code, despite all 5 tools being read-only operations. The filter objects are deeply nested with repetitive EQ sub-objects, increasing cognitive load and token cost without adding semantic value. Overall, the server delivers functional CRUD tool discovery but lacks the polish and error guidance expected of production-grade agentic tools.
Fetch classifications from the AIID GraphQL API. Supports filtering, pagination, sorting, and selecting specific classification fields.
Fetch incidents from the AIID GraphQL API. Supports filtering, pagination, sorting, and selecting specific incident fields.
Fetch reports from the AIID GraphQL API. Supports filtering, pagination, sorting, and selecting specific report fields.
Fetch the GraphQL schema from the AIID GraphQL API in JSON format.
Fetch taxonomies from the AIID GraphQL API. Supports filtering, pagination, sorting, and selecting specific taxonomy fields.
No output schemas documented. Agents cannot plan downstream calls without discovering response structure at runtime. Tools must state what fields are returned and their types.
Numeric parameters (pagination.limit, pagination.skip) lack min/max constraints. Agents cannot know valid ranges and may pass absurd values (limit=999999 or negative skip).
Filter parameter structure is verbose and unintuitive. All filters use nested EQ sub-objects (e.g., incident_id: { EQ: 123 }), adding cognitive load and token cost without semantic benefit. Simplify to direct equality: incident_id: 123.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 50 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 44 | - | v1 |
No error handling guidance. Tools do not document retryable vs fatal errors, or what recovery steps agents should attempt (e.g., 'API rate-limited, retry after 60 seconds').
No tool annotations present. All 5 tools are read-only (GET operations), but readOnlyHint annotation is not visible in server.tool() calls, forcing agents to infer safety.
Parameter descriptions do not clarify when to use similar tools. E.g., get-classifications vs get-taxonomies, agents cannot infer the semantic difference from descriptions alone.
get-schema tool lacks usage guidance. Description does not state when to call it (e.g., 'Call first to discover available query types and fields') or explain the format of the returned schema.