Loading ToolBench
Declarative, governed agents on BoundFlow — objective, tools, and policy as YAML. Charter is a control plane for agents that run on BoundFlow, managing agent configuration, MCP server integration, and policy enforcement.
Charter exhibits significant quality gaps across naming, descriptions, and schemas. Tool definitions are duplicated (get_ticket, get_charge, create_refund, hangs appear 2-3 times each), indicating inconsistent registration or test fixtures mixed with production tools. Descriptions vary wildly in quality: some are adequate (search_tickets: 'Every open ticket, newest concern first'), others are vague or test-oriented (hangs: 'Never answers. For exercising the tool timeout'). Input schemas are present but minimal, most parameters lack detailed constraints, ranges, or format guidance. Error handling is absent from tool definitions. The server mixes production support tools with test/playground fixtures (always_fails, undeclared_danger), which violates single-responsibility and composition principles.
Fails every time. For watching max_tool_failures trip and name the tool.
Close a ticket.
Refund a charge. Money moves — this is the one a human approves.
Refund against a charge. A person approves this before it runs.
Look up one charge.
What was charged, and what has already been refunded against it.
Fetch a support ticket by id, with the charge ids it refers to.
Tool duplication and inconsistent registration: get_ticket, get_charge, create_refund, and hangs are defined multiple times across examples/support_server.py, playground/mcp_server.py, and tests/mcp_fixture_server.py. This violates single-responsibility and composition principles, confuses LLM tool selection, and suggests test fixtures are exposed as production tools.
Test and playground tools exposed in production: always_fails, undeclared_danger, and hangs are test fixtures with descriptions like 'For watching max_tool_failures trip' and 'Never answers. For exercising the tool timeout.' These should not be registered in a production MCP server.
Minimal parameter descriptions and missing constraints: Parameters like ticket_id, charge_id, and amount_usd lack format guidance, ranges, or validation rules. E.g., create_refund's amount_usd has no min/max bounds, allowing LLMs to pass invalid values. Descriptions should state 'The refund amount in USD (0.01 - charge amount, 2 decimals)'.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | 2026-07-28+ | v2 |
Fetch a support ticket by id.
One ticket, including the charge behind it.
Never answers. For exercising the tool timeout.
Takes longer than any sane timeout. For watching what a hung server does. A server that hangs is the failure `max_tool_seconds` exists for, and it is a different shape from one that errors: nothing comes back at all, so there is no error result for anyone to classify.
Every ticket still open. How a scheduled agent finds its own work.
Every open ticket, newest concern first.
A tool no agent config declares.
No output schema documentation: Tool definitions do not document return types or field structures. E.g., search_tickets returns 'list[dict]' but does not specify which fields (ticket_id, subject, charge_id, etc.) are guaranteed. LLMs cannot plan downstream calls without knowing response structure.
Missing error handling and recovery guidance: No tool definition includes error cases, retryability, or recovery steps. E.g., create_refund does not document what happens if the charge_id is invalid, if the refund exceeds the charge amount, or how the LLM should respond. Descriptions should include 'Returns error if charge_id not found or refund exceeds charge amount.'