Loading ToolBench
Production reference implementation for multi-agent orchestration with MCP contracts, DAG orchestration, and production resilience patterns
This server demonstrates foundational tool structure but falls significantly short of production quality. While 13 tools are defined with basic schemas and descriptions, critical gaps undermine usability: (1) Naming lacks consistency, tools like 'market_data', 'risk_model', 'rebalancer' do not follow verb_noun convention (should be 'fetch_market_data', 'compute_risk_model', 'generate_rebalance_recommendations'). (2) Descriptions are present but generic and brief (avg ~60 chars, well below the 194-char baseline for A+ tools). For example, 'Fetches current market prices and volume for given tickers' lacks context on when to use it vs alternatives, expected latency, or failure modes. (3) Parameter descriptions are entirely absent, the 'tickers' array, 'period' string, and 'prices' object have no inline documentation explaining format, constraints, or valid values. (4) Output schemas are documented in code but lack descriptions for return fields. (5) No error handling guidance, tools do not indicate what errors are retryable, what recovery steps exist, or how to handle partial failures. (6) No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear risk distinctions (WRITE vs READ_ONLY vs REVERSIBLE). The financial agent example shows promise (circuit breaker, token budgeting) but tool definitions themselves are underdeveloped.
Formats the synthesised summary as a final user-facing answer.
Charge the customer's credit card.
Sends the research summary by email (side-effecting).
Fetches current market prices and volume for given tickers.
Decomposes a research query into actionable search terms.
Generates trade recommendations to rebalance the portfolio.
Release locked stock — compensates ReserveInventory.
Tool names do not follow verb_noun convention. 'market_data', 'risk_model', 'rebalancer', 'planner', 'summarise', 'answer' lack action verbs. LLMs infer intent from names, 'fetch_market_data', 'compute_risk_model', 'generate_rebalance_recommendations' are self-documenting.
Parameter descriptions are completely absent. 'tickers', 'period', 'prices', 'query', 'order_id', 'var_95', 'portfolio_value', 'risk_flags', 'search_terms', 'original_query', 'results', 'query_used', 'summary', 'recipient' have no inline documentation. LLMs cannot infer format, constraints, or valid values without descriptions.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 38 | <=2025-11-25 | v2 |
Lock stock for an order.
Computes a simplified Value-at-Risk estimate for the portfolio.
Send order confirmation email.
Synthesises raw search results into a structured summary.
Executes a web search and returns ranked result snippets.
Search the web for a query.
Tool descriptions are generic and brief (avg ~45 chars vs 194-char baseline). 'Fetches current market prices and volume for given tickers' lacks context on when to use it, expected latency, failure modes, or dependencies. Descriptions should answer: What? When? Why? What does it return?
No tool annotations despite clear risk distinctions. Tools marked WRITE (charge_card, reserve_inventory, send_notification, email_dispatch) and REVERSIBLE (release_inventory) lack destructiveHint/idempotentHint annotations. Agents need to know which calls are safe to retry and which have irreversible consequences.
No error handling guidance. Tools do not indicate retryable vs fatal errors, recovery steps, or partial failure modes. E.g., 'charge_card' failing should guide the agent to call 'release_inventory' for compensation, but no such guidance exists.