Loading ToolBench
A Model Context Protocol (MCP) server for integrating Billforward's billing API with LLMs.
Billforward MCP has mixed quality. Tools are explicitly registered with names, descriptions, and input schemas using Zod validation. Most descriptions are moderately detailed (50-150 chars) and explain both the purpose and constraints. However, several critical gaps exist: (1) No output schemas are documented, users cannot predict what fields to expect from responses, forcing LLMs to infer structure. (2) Parameter descriptions are inconsistent, some tools like 'find-accounts' explain constraints clearly, while 'search-tools' provides minimal guidance on what 'query' should contain. (3) Error handling is basic (isError flag + text message) with no recovery guidance, LLMs cannot self-correct. (4) Tool naming is mostly clear (verb_noun style), but some names like 'search-tools' and 'describe-tool' suggest internal implementation details rather than user intent. (5) STDIO transport caps protocol readiness at 50, and the server provides no deprecated-pattern usage to migrate, making this a conservative mid-range score.
Run a catalog operation by exact name from search-tools (e.g. list-payments, list-credit-notes-by-account). Writes need a write-enabled environment. Prefer lookup for single prefixed IDs.
Return the full catalog entry for one API operation (method, path template, params). Use the exact name from search-tools.
Find accounts by exactly one of: account ID, exact profile email, or metadata filter. NOT for company/hotel names or free text — use lookup instead.
Read one account plus its subscriptions and invoices. Does NOT include credit notes or payments — use search-tools → call-api (list-credit-notes-by-account / list-payments) for those.
Customer 360 by email or ACC- id: account, subscriptions (state/dunning), credit balances, recent invoices. Use for active-sub / dunning panoramas. Does NOT list credit notes or payments — use search-tools → call-api for those.
No output schemas documented for any tool. LLMs cannot predict response structure (fields, types, nesting) and must infer from examples or hallucinate. This violates pattern:response-shaper and increases hallucination risk.
Error handling is minimal: only isError boolean + unstructured text message. No categorization of error type (retryable, user-fixable, fatal), no recovery guidance (e.g., 'try search-tools first'), no invalid value/constraint reporting. Violates pattern:recovery-guide.
Parameter 'query' in search-tools lacks specificity. Description says 'Search query' with minLength=1, but does not explain: is this free-text or structured (e.g., 'list-*', 'get-payments')? What operators are supported? LLMs cannot infer optimal query syntax.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 58 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 59 | - | v1 |
Validate API credentials and get information about the authenticated user/organization.
Analyzes recently used metadata keys across Accounts, Subscriptions, and Invoices to provide a 'cheat sheet' of available custom fields (e.g., salesforce_account_id, duns, payment_gateway_url). Use this to discover how to filter using the 'metadata' parameter.
Short guide: which Billforward MCP tool to use (lookup vs catalog vs customer summary) and configured environments.
Prefer this for any prefixed Billforward ID (ACC-, SUB-, INV-, PAY-, REC-, CDT-, …) or free-text search (company name, email, Hotel Mio, etc.) via /search-v3. Do not use find-accounts for company names. Pass resource= only for UUID-only entities.
Search the Billforward API catalog when no always-on tool fits (lists, writes, credit-notes-by-account, payments, receipts, etc.). Returns operation names — then describe-tool (optional) and call-api. Not for single-ID lookup (use lookup).
Tool names 'search-tools', 'describe-tool', 'call-api' reflect implementation (discovery, introspection, execution) rather than user intent. Names like 'search_catalog', 'get_operation_details', 'execute_operation' would be more descriptive and help LLMs select the right tool.
'call-api' accepts 'args' as generic object with no schema or documentation of expected structure. Response from call-api is also unstructured JSON. Without knowing which Billforward API endpoint is invoked, downstream tools cannot depend on specific fields. This breaks pattern:tool-chain.
No pagination guidance in list/search tools. Tools like search-tools, find-accounts, get-account-history, lookup all accept 'limit' and 'offset' parameters, but descriptions do not explain: is there a total count returned? Can offset exceed total? What does a zero-result response look like? Violates pattern:paginated-result.