Loading ToolBench
Async Python web framework where one route is an HTTP endpoint, an OpenAPI operation and an MCP tool
Veloce exposes 3 meta-tools (search_tools, describe_tools, run_tools) that form a dynamic tool discovery and execution system. All three have descriptions and input schemas present. However, schemas lack detailed parameter descriptions and output schemas are not documented. The run_tools tool accepts a complex 'steps' array but provides no guidance on step structure, error handling, or chaining semantics. Descriptions are adequate but generic (194-char baseline suggests room for improvement). No tool annotations (readOnlyHint, destructiveHint, idempotentHint) are present despite run_tools being a WRITE operation. Error handling is mentioned but not detailed in tool descriptions.
Return the full definition - description, input schema, annotations - of each named tool, which is what is needed to call it. Names come from search_tools.
Run several tool calls in one request. Each step names a tool and its arguments; an argument of the form {"$from": "<step id>", "path": "<JSON pointer>"} is replaced by that part of an earlier step's result, so a chain of calls costs one round trip. Set quiet on a step whose result is only needed by a later step and it is left out of the response. Stops at the first failing step unless stop_on_error is false.
Find the tools that fit a task. Returns the best matches by name, with a one-line description each; call describe_tools for a match's full definition before calling it. This server publishes its catalogue through search rather than listing it, so this is where every other tool is found.
run_tools lacks output schema documentation. Callers cannot predict the structure of results, making chaining unreliable. No guidance on per-step success/failure format or error propagation.
run_tools input schema ('steps' array) has no description of step structure, required fields, or valid tool names. LLMs cannot construct valid step objects without trial-and-error.
No tool annotations present. run_tools is a WRITE operation (can execute arbitrary tools) but lacks destructiveHint. search_tools and describe_tools lack readOnlyHint. Agents cannot reason about safety.
Parameter descriptions are minimal. 'query' in search_tools and 'names' in describe_tools lack guidance on format, length, or expected values. 'stop_on_error' in run_tools lacks explanation of default behavior.
Inferred effective spec: 2026-07-28+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 54 | 2026-07-28+ | v2 |
Error handling not documented in tool descriptions. No guidance on what errors are retryable, what indicates user misconfiguration, or how to recover from partial failures in run_tools.