Loading ToolBench
Agent-first FX / hedging micro-execution API (simulated venue). Declare a currency exposure, request a quote, then execute it; every fill comes with a full, reconciling cost attribution. Execution is simulated — no real money moves.
Numera demonstrates solid definition quality with 12 well-named tools following verb_noun conventions (declare_exposure, get_quote, execute_hedge). All tools have descriptions (avg ~80 chars, within baseline 34-392 range). Input schemas are present and typed for all tools. However, descriptions are terse (many under 100 chars) and lack LLM-optimized guidance on WHEN to use each tool or dependencies between them. Parameter descriptions are minimal, many lack format/constraint details. Output schemas are inferred from DTO classes but not explicitly documented in tool definitions. Error handling uses _translate_errors() context manager to surface domain errors as ToolError with stable codes, which is good, but recovery guidance is absent. No tool annotations (readOnlyHint/destructiveHint/idempotentHint) despite clear risk classifications (WRITE vs READ_ONLY). Composition is strong: tools chain well (declare → quote → execute → order), and idempotency is supported (execute_hedge via idempotency_key). Security: no secrets in params, but no explicit permission gates or audit scope declarations visible.
Sign off on an order parked above the approval threshold, then execute it.
Declare a currency exposure; returns the exposure and the convert/hedge decision.
Execute an accepted quote. Idempotent on ``idempotency_key`` (safe to retry).
Query the append-only audit trail (filter by agent, subject, event type).
Fetch the itemised cost breakdown for a filled order (reconciles to the all-in cost).
Fetch an order with its fill and cost attribution.
Net FX exposure per currency for an agent, with aggregate valued in the reference ccy.
Descriptions lack LLM-optimized guidance. Most tool descriptions are 55-70 chars, below the 194-char baseline for A+ tools. Missing WHEN to use, dependencies, and recovery hints. E.g., 'get_quote' should explain 'Call after declare_exposure; quote expires after TTL seconds.'
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear risk classifications. Tools marked WRITE (declare_exposure, execute_hedge, set_policy, approve_order) should carry destructiveHint=true. Tools marked READ_ONLY should carry readOnlyHint=true. execute_hedge should carry idempotentHint=true (safe to retry on idempotency_key).
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 56 | <=2025-11-25 | v2 |
Request a time-bounded quote (with TTL) for a declared exposure.
Per-agent report: realized cost, outstanding exposure, and order counts by status.
Revalue a filled order at the current mid; returns unrealized P&L (target currency).
Reconcile an order: venue status + double-entry ledger balance per currency.
Set an agent's risk mandate. Caps are minor units in ``reference_currency``.
Parameter descriptions lack format/constraint details. E.g., 'direction' and 'timing' are enums but descriptions do not list valid values (HAVE|OWE, SPOT|FORWARD). 'value_date' lacks ISO 8601 format hint. 'allowed_pairs' and 'allowed_instruments' lack examples or format guidance.
Output schemas not explicitly documented in tool definitions. Tool descriptions do not state what fields are returned or their types. LLMs must infer structure from DTO class names (ExposureDecisionResponse, QuoteResponse, etc.), which are not visible in the tool definition itself.
Error handling surfaces domain errors with stable codes (e.g., '[EXPOSURE_NOT_FOUND] ...') but lacks recovery guidance. Descriptions do not explain what errors are possible or what the LLM should do next (retry, ask user, call a different tool).