Loading ToolBench
Elixir framework for building production-ready AI agents and LLM applications: type-safe schemas with JSON Schema generation, tool calling, streaming, multi-agent coordination, guardrails, and distributed fault-tolerant sessions, with first-class Anthropic Claude support.
Normandy is an Elixir framework providing agent orchestration and tool coordination infrastructure. The evaluation focuses on the 6 example tools provided (3 stubs + 3 customer support tools). Tools have basic schemas with type definitions and descriptions present, but lack depth in LLM-optimization, parameter documentation completeness, and error handling guidance. The tools exhibit inconsistent naming conventions (stub_search vs knowledge_base_search) and descriptions vary widely in actionability. Schemas are present but minimal, most lack output documentation. No evidence of tool annotations (readOnlyHint/destructiveHint), pagination support for list-like tools, or recovery guidance in error cases.
Create a support ticket for issues requiring follow-up or escalation. Returns a ticket ID that can be used to track the issue.
Search the knowledge base for information about products, policies, troubleshooting, and common questions. Optionally filter by category.
Look up order information by order ID. Returns order status, items, tracking number, and estimated delivery date if available.
Process a refund request for an order. Validates eligibility and initiates the refund. Can process full or partial refunds.
Offline stub scrape tool.
Offline stub search tool.
Stub tools (stub_search, stub_scrape) have empty input schemas with no parameters and minimal descriptions. These appear to be test fixtures, not production tools, yet are included in the evaluation set. Descriptions do not explain when to call them vs real tools.
No tool descriptions include explicit guidance on error recovery or dependencies. E.g., knowledge_base_search does not state: 'If no results found, try a broader query or call category list first.' Order_lookup lacks guidance on what to do if order not found (suggest search? escalate?).
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract expected fields. E.g., order_lookup says it 'Returns order status, items, tracking number, and estimated delivery date' but the actual JSON response structure is not defined.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 50 | 2025-06-18+ | v2 |
process_refund (WRITE operation) lacks any confirmation/dry-run pattern. An agent could irreversibly refund an order without verification. Descriptions mention 'validates eligibility' but do not expose pre-check or confirmation step.
Tool names lack consistent verb-prefixes in some cases. 'stub_search' and 'stub_scrape' use underscore-prefix for stub namespace rather than semantic verbs. 'knowledge_base_search' vs 'order_lookup' mismatch: one uses search_, one uses lookup. Inconsistency forces LLM to reason about name patterns.
process_refund's 'amount' parameter is optional and defaults to 'full order amount' per description, but no JSON schema default is visible. Undeclared defaults can cause silent incorrect behavior if LLMs omit the parameter intentionally.
No tool annotations present (readOnlyHint, destructiveHint, idempotentHint). This prevents MCP clients from rendering safe/unsafe warnings or disabling retry on destructive operations. process_refund should be marked destructive; knowledge_base_search should be readOnly.
Knowledge_base_search's 'category' parameter is optional with enum values, but description does not explain what happens if category is omitted (search all? default category?). Undocumented behavior forces LLMs to guess.
No pagination or result-limiting strategy visible in schemas. If knowledge_base_search returns 100+ results, context window bloats. No limit, offset, or pagination_token parameters declared.