Loading ToolBench
MCP server for Crayon CloudIQ cost and billing APIs with HTTP transport
The server defines 28 tools with explicit schemas and descriptions. All tools are properly registered with JSON Schema input definitions and non-empty descriptions. However, there are systematic gaps in parameter-level documentation and output schema definition. Most tool descriptions are moderate length (50-150 chars) and explain what the tool does, but many lack guidance on WHEN to use it or dependency hints. Parameter descriptions exist but are often minimal (10-30 chars). No tool descriptions document the output schema structure, forcing LLMs to infer response shapes. Error handling is not visible in the definitions. Security-wise, the server uses environment variables for credentials (dotenv pattern seen in index.ts) rather than tool parameters, which is correct. Naming is verb_noun consistent (get_*, update_*) and generally clear. The single WRITE operation (update_subscription_tags) properly declares its risk. Overall the server is solidly mid-tier: definitions are present and structured, but lack the polish and LLM-optimization of top-tier toolkits.
Analyze cost distribution across tags to identify top cost drivers by tag.
Detect unusual cost spikes and anomalies in billing data. Returns anomalies with severity ratings.
Find subscriptions and invoices matching a name pattern using regex. Useful for identifying resource groups.
Get Azure costs for a specific date range with detailed breakdown.
Get Azure costs broken down by subscription for a specific date range.
Get details about an Azure Plan including subscription count and billing configuration.
Output schemas not documented. Tools return data but the structure is never defined in tool definitions. LLMs cannot plan what fields to expect or compose downstream calls. Example: get_billing_statements description says 'Returns monthly billing data' but never specifies the response structure (e.g., does it return {data: [...], total: number} or flat array?).
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 60 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 47 | - | v1 |
Get all Azure subscriptions for a specific Azure Plan.
Get Azure subscriptions for a customer tenant. Use this to correlate costs with specific Azure resources.
Get detailed Azure usage data for a specific subscription and time period. Returns a SAS URI to download CSV file with usage details.
Get billing statements for an organization with optional filters. Returns monthly billing data including total sales prices and invoice profiles.
Get detailed cost breakdown by subscription with resource correlation. Combines billing data with subscription details.
Analyze cost trends over time to identify growth patterns and forecast future costs.
Get customer tenants (Azure/AWS customers) for resource correlation. Use this to discover tenant IDs for subscription queries.
Get grouped billing statements by billing cycles for an organization. Useful for aggregated cost analysis.
Get historical billing data over multiple months. Useful for cost trend analysis and forecasting.
Get invoice profiles for an organization. Invoice profiles are used to group subscriptions for billing purposes.
Get invoices for an organization. Returns invoice details including amounts, dates, and status.
Get last month's costs grouped by invoice profile.
Get last month's costs for an organization.
Get last month's costs aggregated by subscription tags.
List all organizations accessible with current credentials. Use this to discover organization IDs for other queries.
Get detailed information about a specific subscription including resource counts and billing status.
Get tags for a subscription. Tags are used for cost allocation and resource organization.
Get all cloud subscriptions (Azure, AWS, etc.) to correlate with billing data and resources.
List all subscriptions with their tags for comprehensive resource inventory.
Get cost breakdown by subscription tags over time. Useful for tracking costs by cost center or project.
Update tags for a subscription. Use for cost center allocation and resource grouping.
Generate a pie or doughnut chart visualizing cost distribution by subscription. Returns chart as base64 data URI.
Many parameter descriptions are trivially short (10 - 20 chars). Examples: get_organizations has empty properties description, get_invoice_profiles description is 'Invoice Profile ID (required)' without context. Descriptions should explain format constraints, enums, ranges, and when to use each param.
Tool descriptions lack 'WHEN to use it' guidance and dependency hints. E.g., get_organizations says 'List all organizations...' but does not explain when to call this first, or that it returns IDs needed by other tools. Many tools are interdependent (e.g., get_organizations → get_billing_statements) but this is not documented, forcing LLMs to discover patterns by trial.
Tool naming: 'find_similar_subscriptions_and_invoices' contains 'and', suggesting it does two things (find subscriptions AND find invoices). Consider splitting into find_subscriptions_by_pattern and find_invoices_by_pattern, or clarify in description that it searches both in one query.
Redundant tool sets suggest missing composition patterns. Example: get_last_month_costs_by_tags, get_last_month_costs_by_organization, and get_last_month_costs_by_invoice_profile are nearly identical (same parameters, just group by different field). This triplicates tool definitions when a single 'get_last_month_costs(organizationId, groupBy?)' with groupBy enum would be cleaner and reduce cognitive load on the LLM.
No error handling guidance visible in tool definitions. Tools do not document what errors can occur, whether they are retryable, or what the LLM should do next. Example: detect_cost_anomalies may fail if organizationId is invalid, but there is no error documentation.
Parameter constraints not consistently declared. Example: detect_cost_anomalies has 'changeThresholdPercent' with range 1 - 100 documented in description, but other numeric parameters (page, pageSize, monthsBack) lack min/max constraints in schema. JSON Schema should include minimum/maximum fields.
Tool descriptions are inconsistently detailed. Some like 'detect_cost_anomalies' (97 chars) and 'visualize_costs_pie_chart' (109 chars) are informative, but others like 'get_organizations' (64 chars) or 'get_invoice_profiles' (88 chars) are minimal.