Loading ToolBench
FastMCP server for Stripe payment gateway, billing ops, customer support, and Austrian/EU tax compliance
stripe-mcp has 8 tools with basic schemas and descriptions, but significant gaps prevent higher scores. Tool names lack clear action verbs (manage_* is vague; should be create_, update_, list_, get_). Descriptions exist but are generic and lack LLM-optimized guidance on WHEN to use each tool vs. alternatives. Parameter descriptions are present but sparse (avg ~40 chars vs. baseline 72). Output schemas are not documented, LLMs cannot predict response structure. Error handling is absent from visible code. Security concerns: no evidence of secret injection for Stripe API keys. Tools 7 - 8 (prefab UI cards) have minimal descriptions and no visible schema definitions in source.
Calculate Austrian VAT rates and verify EU ATU VAT IDs for Reverse Charge eligibility. ## Return Format Returns tax breakdown, gross total, and BAO § 132 compliance metadata. ## Examples - `calculate_austrian_vat(amount=100.00, customer_vat_id="ATU12345678")`
Create Stripe Payment Links, Checkout Sessions, or Invoices with Austrian/EU tax calculation. ## Return Format Returns checkout URL or invoice object with tax breakdown. ## Examples - `manage_stripe_checkout(operation="create_payment_link", amount=99.00)`
Manage Stripe customer records. ## Return Format Returns dictionary with operation results and customer data. ## Examples - `manage_stripe_customers(operation="list")`
Manage Stripe charges, payment intents, refunds, and disputes with safety caps. ## Return Format Returns charge or refund transaction status. ## Examples - `manage_stripe_payments(operation="list_charges")`
Manage Stripe subscriptions. ## Return Format Returns subscription list or detailed item object. ## Examples - `manage_stripe_subscriptions(operation="list")`
Tool names use 'manage_*' prefix, which is vague and does not convey specific actions. Should use verb_noun pattern: list_customers, create_customer, update_customer, etc. LLMs infer intent from names before reading descriptions.
Output schemas are not documented in tool descriptions. LLMs cannot predict response structure, field names, or types. Must document: 'Returns object with fields: {id: string, email: string, created: ISO8601, ...}'
Tools 7 - 8 (show_customer_billing_health, show_revenue_kpi_dashboard) have minimal descriptions (<30 chars) and no visible input/output schemas in source code. Cannot verify schema completeness.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 46 | <=2025-11-25 | v2 |
Prefab UI card showing customer billing health and active subscriptions.
Prefab UI card displaying MRR, active subscriber count, and Austrian VAT totals.
Retrieve Stripe revenue analytics, MRR, churn rate, and Austrian VAT summaries. ## Return Format Returns SaaS financial KPIs and tax collection totals. ## Examples - `stripe_revenue_analytics(metric="all")`
No error handling guidance visible in tool definitions. Descriptions do not explain what happens on failure, whether errors are retryable, or what the LLM should do next. E.g., 'If customer not found, try search_customers() with partial email.'
Stripe API key is set via settings.stripe_api_key in server.py (line: stripe.api_key = settings.stripe_api_key). If settings are environment-based, this is acceptable; if exposed as a tool parameter, it is a critical security leak. Verify secret injection pattern.