Loading ToolBench
MCP gateway that safely exposes a double-entry payments ledger to an AI agent.
Server has 7 tools with mostly complete schemas and descriptions. Naming follows verb_noun convention (get_, create_, refund_, check_). Descriptions are present and substantive (100-300 chars), explaining WHAT the tool does and key constraints (idempotency, approval gates, scopes). Schemas include proper types, descriptions, and constraints (min/max for amounts/limits). However, several tools lack output schema documentation, responses are inferred from context, not formally specified. Error handling is mentioned in descriptions but not structured in tool definitions. Tool composition is sound (single responsibility, proper chaining via IDs). Security-relevant details (scopes, approval gates) are documented in descriptions but not formalized as tool annotations beyond the basic readOnlyHint/destructiveHint.
Admin-only: assert the ledger is balanced (net 0, no drifted accounts). Requires admin scope. Returns only the integrity summary — never raw account data.
Create a payment moving money from one account to another. Idempotent: retrying the same logical operation (same operation_id, same arguments) replays the original result instead of charging twice. Amounts over the agent's autonomous limit return an APPROVAL_REQUIRED response and are NOT executed.
Return a bounded page of an account's most-recent postings (newest first).
Return the current balance (in minor units) of an allowed account.
Fetch a single payment by id. Read-only. Returns the payment or a structured error.
Health check. Returns 'pong' if the payments MCP gateway is alive.
Output schemas not formally documented. Tool descriptions mention what is returned (e.g., 'Returns the payment or a structured error') but JSON Schema output types are not visible in the source. LLMs cannot plan downstream tool calls without knowing response field names and types.
Scope/permission requirements mentioned in descriptions but not formalized as tool annotations or metadata. 'Admin-only' and 'requires refund scope' are text hints; no machine-readable scope declaration. Agents cannot determine least-privilege configuration.
Error handling guidance is implicit in descriptions ('returns APPROVAL_REQUIRED response') but not structured. No explicit error classification (retryable vs user-fixable vs fatal) or recovery hints in tool definitions. Agents must infer error semantics from text.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 59 | 2026-07-28+ | v2 |
Refund (part of) a payment. More consequential than a payment — it reverses money — so it needs the refund scope. Idempotent by operation_id; amounts over the agent's limit return APPROVAL_REQUIRED and are NOT executed.
Pagination for get_account_ledger is present (limit param, 1-100 range) but no 'next_cursor' or 'total_count' documented in output. LLMs cannot determine if more results exist or how to fetch the next page.
Mandate parameter (nullable string) in create_payment and refund_payment lacks format/constraint documentation. What is the expected format of a mandate? How is it validated? Description says 'Signed mandate' but does not specify encoding, signature algorithm, or validation rules.