Loading ToolBench
City of Carlsbad GIS — Carlsbad, California open geospatial data (ArcGIS). Provides access to parcels, zoning, public works & city services datasets.
Three read-only tools with solid naming (verb_noun pattern) and complete input schemas. Descriptions are present and moderately detailed (120-180 chars), explaining what each tool does and when to use it. However, output schemas are entirely undocumented, LLMs cannot predict response structure. Parameter descriptions are good but lack explicit constraints (e.g., 'limit 1-50' stated in description but not as JSON Schema min/max). No error handling guidance. Missing tool annotations (readOnlyHint). Composition is sound: search_datasets → query_layer/layer_info chain is clear.
Get an ArcGIS Feature/Map Service layer's schema by url: fields (name + type), geometry type, total record count, and capabilities.
Query an ArcGIS Feature Service / Map Service layer by its url (from search_datasets). SQL-like `where`, comma-separated `out_fields`, `order_by`, `limit`, `offset`. Returns attribute rows (and geometry). Use where="1=1" + out_fields="*" to sample.
Search City of Carlsbad GIS open geospatial datasets (parcels, zoning, public works & city services) by keyword. Returns each dataset's name, summary, record_count, owner/org, and its Feature Service `url` — pass that url to query_layer / layer_info.
Output schemas completely undocumented. LLMs cannot predict response structure (fields, types, nesting). search_datasets returns {total, count, datasets[{id, name, summary, org, org_id, record_count, type, url}]} but this is invisible to the agent.
No tool annotations. readOnlyHint missing from all three tools. Agents cannot distinguish safe (read-only) from destructive operations without explicit hints.
No error handling guidance. If query_layer receives an invalid where clause or layer_info hits a 404, the tool throws a bare error message ('ArcGIS query: 404'). No recovery hints for the agent.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | 2026-07-28+ | v2 |
Parameter constraints stated in descriptions but not formalized in JSON Schema. 'limit 1-50' is prose; should be {type: number, minimum: 1, maximum: 50}. LLMs cannot parse prose constraints reliably.
No pagination metadata in search_datasets response. If total > 20 (default limit), agent cannot fetch remaining datasets. Missing next_cursor or offset guidance.