Loading ToolBench
California State Parks open geospatial data (ArcGIS) — search datasets, query layers, and retrieve layer schema information.
Three well-named, read-only tools with clear descriptions (avg 180 chars, within baseline 194). All tools have input schemas with type definitions and required fields. However, output schemas are completely undocumented, LLMs cannot predict response structure for chaining. Parameter descriptions are present but lack constraints (e.g., 'limit' has no min/max stated in description, only in code). No error handling guidance. Missing tool annotations (readOnlyHint). Composition is sound: search_datasets → query_layer/layer_info chain is logical.
Get an ArcGIS Feature/Map Service layer's schema by url: fields (name + type), geometry type, total record count, and capabilities.
Query an ArcGIS Feature Service / Map Service layer by its url (from search_datasets). SQL-like `where`, comma-separated `out_fields`, `order_by`, `limit`, `offset`. Returns attribute rows (and geometry). Use where="1=1" + out_fields="*" to sample.
Search California State Parks GIS open geospatial datasets (parks, trails, recreation & natural-resource layers) by keyword. Returns each dataset's name, summary, record_count, owner/org, and its Feature Service `url` — pass that url to query_layer / layer_info.
Output schemas completely undocumented. LLMs cannot predict response structure (fields, types, nesting) for downstream chaining or data extraction. search_datasets returns {total, count, datasets[{id, name, summary, org, org_id, record_count, type, url}]} but this is invisible to the LLM.
Parameter constraints not stated in descriptions. 'limit' in search_datasets says '1-50, default 20' but only in description text, not as JSON Schema minumum/maximum. query_layer 'limit' says '1-2000, default 50' but lacks formal constraint. LLMs may pass invalid values; validation happens at runtime only.
No error handling guidance. If query_layer receives an invalid URL or ArcGIS returns 404, the tool throws 'ArcGIS query: 404' with no recovery hint. LLM cannot determine if it should retry, ask user, or try a different layer.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 59 | 2026-07-28+ | v2 |
Missing tool annotations. All three tools are read-only (no side effects), but readOnlyHint is not declared in tool definitions. Agents cannot distinguish safe tools from destructive ones without explicit hints.
query_layer 'where' parameter accepts arbitrary SQL. No validation or injection guard documented. Description says 'SQL where clause' but does not warn about SQL injection risk or sanitization. Agents may pass untrusted input.