Loading ToolBench
Universal MCP server for querying GraphQL, REST, and gRPC APIs using natural language
Ratatoskr provides 9 tools with basic schema and description coverage, but multiple critical gaps prevent a higher score. All tools have input schemas visible in api_agent/tools.py with proper JSON Schema types (string, object). Descriptions are present for all tools but most are terse (10-50 chars) and lack actionable context about WHEN to use each tool, WHAT gets returned, or how they compose. Parameter descriptions are minimal, 'Query string', 'HTTP method', 'Request message' lack format/constraint guidance. No tool documentation of return schemas, pagination, limits, or error recovery paths. No enum constraints visible on the 'method' parameter (GET, POST, PUT, DELETE, PATCH) or 'api_type' parameter (graphql, rest, grpc). No evidence of idempotence hints, destructive operation warnings, or per-tool security scopes. The server uses fastmcp (HTTP transport), which is current, but provides NO tool annotations (readOnlyHint/destructiveHint), no explicit error guidance, and no pagination support for list_stored_schemas or list_recipes. Tools like query_rest and query_grpc are marked WRITE risk but have no dry-run, confirmation, or rate-limit guards in visible code.
Get schema and capabilities of an API
Execute a saved recipe with parameter values
List saved API query recipes
List all stored API schemas
Execute GraphQL queries against remote APIs
Execute gRPC service method calls
Execute REST API requests
Save a reusable API query recipe
Tool descriptions are terse (10 - 50 characters) and lack actionable context. Missing: WHEN to call (vs similar tools), WHAT gets returned, prerequisites, and dependency hints. E.g., 'Execute REST API requests' tells an LLM nothing about retry behavior, rate limits, or timeout handling.
No enum constraints visible on parameters that accept a finite set of values. 'method' parameter (GET, POST, PUT, DELETE, PATCH) and 'api_type' parameter (graphql, rest, grpc) lack enum declarations. LLMs will hallucinate invalid values.
No output schema documentation visible. Tools return unstructured data without clear field definitions. 'query_rest' and 'query_graphql' responses are opaque, LLMs cannot plan downstream tool calls or extract required fields (e.g., pagination cursors, chaining IDs).
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 53 | <=2025-11-25 | v2 |
Store and cache API schema for later use
Destructive tools (query_rest with POST/PUT/DELETE, query_grpc, store_schema, save_recipe) lack dry-run or confirmation workflows. No evidence of idempotent hints, permission checks, or rate-limiting. Agents can accidentally trigger multiple identical mutations.
Parameter descriptions are incomplete. 'Query string', 'HTTP method', 'Request message' do not specify format, length, or constraint rules. LLMs cannot infer whether a URL must be absolute, a query must be under 4KB, or headers must be a flat dict.
No tool annotations (readOnlyHint/destructiveHint/idempotentHint) visible. LLMs cannot distinguish read-only discovery tools from write operations. 'list_recipes' and 'list_stored_schemas' are marked READ_ONLY but have no formal annotation.
'list_stored_schemas' and 'list_recipes' have empty input schemas but no pagination parameters (limit, offset, next_cursor). Without pagination, these tools cannot scale beyond a few hundred items. LLM context will be flooded by large responses.
No error guidance visible. Tools do not document what errors can occur, how to retry, or when to ask the user. A failed 'query_rest' call likely returns raw HTTP status or exception, LLMs will not know whether to retry, fix input, or escalate.