Loading ToolBench
A local emulator for enterprise SaaS APIs, served over your own corpus with per-document ACLs
Backlot exposes 2 tools for GraphQL introspection and pagination handling. Both tools have descriptions (10-20 chars baseline met), but critical gaps undermine quality: (1) Input schemas are present but lack proper type definitions and parameter descriptions, 'introspection' and 'tool'/'arguments' are typed as generic objects with minimal guidance; (2) No output schemas documented, LLMs cannot predict return structure or chain results; (3) Tool names lack action verbs ('derive_tools' is vague; 'with_page_default' is not verb-noun); (4) Parameter descriptions are minimal and do not explain constraints, formats, or valid values; (5) No error handling guidance, what happens if introspection is malformed or depth is negative?; (6) Descriptions are under 200 chars but lack WHEN/WHY context for LLM selection. These are utility/meta-tools for GraphQL handling, not domain tools, which limits their standalone value.
Turn an introspection response into one Tool per root Query field. introspection is the endpoint's reply to INTROSPECTION_QUERY, with or without its data envelope. Tools come back in schema order, which is the order the SDL declares.
arguments with a page size filled in, unless the caller did its own paging. An unpaged list call otherwise gets the server's default page — which is not everything, but is far more than an agent needs, and the measured agent narrows with a filter rather than a page size.
Tool names lack action verbs. 'derive_tools' and 'with_page_default' do not follow verb_noun convention (get_, create_, update_, etc.). LLMs cannot infer intent from names alone.
Input schemas use generic object types without field-level descriptions. 'introspection' is typed as object but lacks guidance on structure, required fields, or format. 'tool' and 'arguments' similarly underdocumented.
No output schemas documented. LLMs cannot predict return structure, field names, or types. Breaks tool chaining and forces agents to guess downstream parameter mappings.
Parameter descriptions are minimal (under 50 chars) and lack constraints. No guidance on valid ranges for 'depth', format of 'introspection' payload, or error conditions.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 41 | 2026-07-28+ | v2 |
No error handling guidance. Descriptions do not explain what happens if introspection is malformed, depth is negative, or tool/arguments are invalid. LLMs have no recovery path.