Loading ToolBench
Operations database for a small fulfillment business: orders arriving from sales channels, production state per item, and FIFO material stock. Order status and stock balance are both DERIVED — move items and record movements, never try to write a status or a balance.
Strong tool definitions with excellent descriptions (avg 250+ chars, well above baseline 194). All 11 tools have clear, LLM-optimized docstrings explaining WHAT, WHEN, and WHY. Input schemas are visible and typed. However, output schemas are not formally documented in the code, return types are inferred from docstrings only. Parameter descriptions are comprehensive but lack explicit enum constraints in schema (e.g., 'status' values stated in text, not as JSON Schema enums). No tool annotations (readOnlyHint/destructiveHint) present. Error handling guidance is implicit in descriptions but not formalized.
Recent mutations, newest first: who changed what, when. Every write in this server is logged with its actor, so agent actions can always be separated from human ones after the fact.
List open orders with their items, most urgent shipping deadline first. This is the default read: call it before answering anything about "what is pending", "what ships today" or "what should we work on". Order `status` is DERIVED from the items and cannot be set directly: open - every item still 'todo' in_progress - some item started ready - every item 'done' The remaining statuses are human decisions set via `order_advance`: packing, shipped, cancelled.
Import a channel order export (CSV) into the database. Idempotent: the same file can be imported any number of times, and overlapping files are expected. Existing orders are updated, never duplicated, and production progress already recorded on the items is preserved across the re-import. Returns a tally of orders created vs updated. If `created` is 0 on a file you believe is new, the orders were already imported — that is normal.
Move one order item through production: 'todo' -> 'doing' -> 'done'. The parent order's status is re-derived immediately and returned, so a single call tells you whether the whole order just became ready.
Output schemas not formally documented. Docstrings describe return types informally ('list[dict]', 'dict') but no structured schema definition visible in code. LLMs cannot reliably parse unstructured output or plan downstream tool calls.
Enum constraints stated in descriptions, not in JSON Schema. E.g., order_advance accepts 'packing'|'shipped'|'cancelled' but schema likely shows type:string without enum constraint. LLMs may hallucinate invalid values.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Tools clearly marked as READ_ONLY or WRITE in metadata, but MCP annotations not visible in code. Agents cannot infer safety/idempotency from tool definitions alone.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 60 | 2026-07-28+ | v2 |
Create or rename a material.
Apply a manual status to an order: 'packing', 'shipped' or 'cancelled'. Refuses transitions that skip production — `ready -> packing -> shipped` is the only path to shipped. Marking an order shipped while items are still unfinished is a data-integrity bug, not a shortcut, so it raises. Use `item_set_state` to move production forward; the order status follows automatically. Do not call this to set 'open', 'in_progress' or 'ready'.
SKUs seen in orders that still have no material recipe, most-sold first. This is the work queue for catalogue maintenance. Filling the top entries gives the largest costing coverage for the least effort — do not attempt to register the whole catalogue at once.
Create or update a SKU and, optionally, its material recipe. A SKU without a recipe is legal and common: orders import fine, only costing is incomplete. Call `pending_recipes` to see which SKUs are worth filling in first.
Consume material oldest-lot-first and return what it really cost. Raises when stock is insufficient rather than going negative or consuming partially — check `stock_status` first if you are unsure. Prefer `item_set_state(..., consume_material=True)` when the consumption belongs to a specific order item; it links the movement to that item.
Register a purchase as a new FIFO lot.
Current stock balance and value per material. `balance` is derived from the movement log (sum of receipts minus consumption), never a stored counter, so it cannot drift. `value` is the remaining FIFO lots priced at what they actually cost.
Error handling guidance is implicit in descriptions but not formalized. E.g., stock_consume 'Raises when stock is insufficient', but no structured error response format or recovery guidance visible in code.
Parameter 'ref' in stock_consume is optional but no default documented. Unclear if omitting it produces a NULL audit trail entry or a sensible default like 'manual'.