Loading ToolBench
City of Richmond GIS — Richmond, Virginia open geospatial data (ArcGIS). Provides access to parcels, zoning, public works & city services datasets through ArcGIS Feature Services.
Three well-named, read-only tools with clear descriptions and complete input schemas. All tools follow verb_noun naming (search_, query_, layer_info). Descriptions are concise (100 - 200 chars) and explain WHAT, WHEN, and output structure. Input schemas include types and required fields. However, output schemas are not documented, LLMs cannot predict response structure for downstream planning. Parameter descriptions lack format constraints (e.g., 'url must be https and end in /FeatureServer/<n>' is in code, not schema). No error handling guidance or recovery hints. Missing pagination documentation for search_datasets despite returning a 'total' field. No tool annotations (readOnlyHint) despite all being read-only.
Get an ArcGIS Feature/Map Service layer's schema by url: fields (name + type), geometry type, total record count, and capabilities.
Query an ArcGIS Feature Service / Map Service layer by its url (from search_datasets). SQL-like `where`, comma-separated `out_fields`, `order_by`, `limit`, `offset`. Returns attribute rows (and geometry). Use where="1=1" + out_fields="*" to sample.
Search City of Richmond GIS open geospatial datasets (parcels, zoning, public works & city services) by keyword. Returns each dataset's name, summary, record_count, owner/org, and its Feature Service `url` — pass that url to query_layer / layer_info.
Output schemas not documented. LLMs cannot predict response structure (fields, types, nested objects) for search_datasets, query_layer, or layer_info. Forces trial-and-error parsing and wastes tokens.
Parameter format constraints embedded in code (layerUrl validation) but not in schema descriptions. LLMs cannot see that 'url' must be https, public host, and match /FeatureServer|MapServer/<n> pattern. Invites invalid calls.
No error handling guidance. Code throws generic errors ('ArcGIS Hub: 404', 'Invalid input') without recovery hints. LLMs cannot self-correct or know whether to retry, ask user, or abandon.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 59 | 2026-07-28+ | v2 |
search_datasets returns 'total' count but no pagination guidance. Description does not explain how to iterate through large result sets or whether limit/offset are supported.
No tool annotations. All three tools are read-only and safe to retry, but lack readOnlyHint annotation. Agents cannot infer safety properties without explicit hints.