Loading ToolBench
Model Context Protocol (MCP) server for Revenium API monitoring, analytics, and AI spending management
This server exhibits severe structural deficiencies across all tools. All 8 tools use an identical generic action-dispatch pattern with vague, underspecified parameter schemas. Tool names are nouns (products, subscriptions, alerts, etc.) rather than action verbs, violating the fundamental pattern:tool convention. Input schemas lack specificity, all declare a generic 'action' enum and a free-form 'parameters' object with no inner schema. This forces LLMs to hallucinate parameter structures without guidance. Descriptions are present but generic and do not explain when to use each tool or what distinguishes them. No documented output schemas exist. No error handling patterns are visible. The server's code references 'ai_routing/tool_registry.py' but the actual tool implementation is not shown, making it impossible to verify error handling, validation, or output shaping. Per-tool scores are uniformly low due to the identical anti-patterns replicated across all 8 tools.
Alert management operations - list, get, create, update, delete alerts
Customer management operations - list, get, create, update, delete customers
Metering management operations - list, get, create, update, delete metering records
Metering elements management operations - list, get, create, update, delete metering elements
Product management operations - list, get, create, update, delete products
Source management operations - list, get, create, update, delete sources
All 8 tools use noun-only names instead of action-verb prefixes. Tool names (products, subscriptions, alerts, etc.) do not convey what action happens when called. LLMs cannot infer intent from the name alone. Per pattern:tool, names should start with verbs: list_products, create_subscription, delete_alert, etc.
All tools declare identical generic input schemas: {action: string, parameters: object} with no inner schema for 'parameters'. The 'parameters' field has no type constraints, required fields, or nested property definitions. This forces LLMs to guess what keys and values belong inside parameters, inviting hallucinated payloads and API errors.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 44 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 0 | 2025-06-18+ | v1 |
Subscription management operations - list, get, create, update, delete subscriptions
Workflow management operations - list, get, start, next_step, complete_step
Descriptions are generic and do not explain when to use each tool or what distinguishes them from each other. 'Product management operations - list, get, create, update, delete products' (86 chars) reads as a feature list, not actionable guidance. Per pattern:tool-description, descriptions should explain WHAT, WHEN, and WHERE in the context, not enumerate actions.
No output schemas are documented. The evaluator cannot determine what fields are returned, their types, or whether they include IDs for downstream tool chaining. Per pattern:tool, callers must know what to expect so they can compose subsequent calls.
Eight tools operating on different resource types (products, subscriptions, alerts, customers, workflows, sources, metering, metering_elements) could be refactored into fewer, more focused tools with explicit action names. The pattern 'list_X, get_X, create_X, update_X, delete_X' repeats across all resources. Consider a single generic 'manage_resource' tool or explicit verb-prefixed tools per action.
The 'action' parameter should be an enum, but no enum constraint is visible in the schema. Without enum validation, LLMs can pass invalid actions like 'remove', 'edit', 'fetch' instead of the intended set (list, get, create, update, delete, or for workflows: start, next_step, complete_step).
No visible error handling patterns. The source code does not show how errors are classified (retryable, user-fixable, fatal), what guidance is returned on failure, or how invalid inputs are reported. Per pattern:recovery-guide, error responses must tell the LLM what to do next.
All tools are marked Risk: WRITE, indicating they can mutate state. There is no mention of permission gates, audit logging, or dry-run/confirmation patterns. Per pattern:permission-gate, destructive operations should verify the caller has authority and log who did what.