Loading ToolBench
City of Roanoke GIS — Roanoke, Virginia open geospatial data (ArcGIS). Provides access to parcels, zoning, public works & city services datasets via ArcGIS Feature Services.
Three well-named, read-only tools with clear descriptions and complete input schemas. All tools follow verb_noun naming (search_, query_, layer_info). Descriptions are substantive (150 - 250 chars) and explain purpose, prerequisites, and return structure. Input schemas include types and descriptions for all parameters. However, output schemas are not documented, LLMs cannot predict response structure. Parameter descriptions lack explicit constraints (e.g., 'limit: 1-50' is stated but not formalized). No error handling guidance or recovery paths. Missing tool annotations (readOnlyHint). Composition is sound: search_datasets → query_layer → layer_info forms a logical chain with proper ID threading (url field).
Get an ArcGIS Feature/Map Service layer's schema by url: fields (name + type), geometry type, total record count, and capabilities.
Query an ArcGIS Feature Service / Map Service layer by its url (from search_datasets). SQL-like `where`, comma-separated `out_fields`, `order_by`, `limit`, `offset`. Returns attribute rows (and geometry). Use where="1=1" + out_fields="*" to sample.
Search City of Roanoke GIS open geospatial datasets (parcels, zoning, public works & city services) by keyword. Returns each dataset's name, summary, record_count, owner/org, and its Feature Service `url` — pass that url to query_layer / layer_info.
Output schemas not documented. LLMs cannot predict response structure (fields, types, pagination). search_datasets returns {total, count, datasets[]} but this is inferred from code, not declared in tool definition.
No error handling guidance. Tools throw generic errors ('ArcGIS Hub: 404', 'Invalid input') without recovery hints. LLMs cannot determine if errors are retryable or user-fixable.
Missing tool annotations. No readOnlyHint declared despite all tools being read-only. Agents cannot distinguish safe tools from destructive ones without explicit hints.
Parameter constraints stated in descriptions but not formalized in schema. 'limit: 1-50' is prose, not a JSON Schema minimum/maximum. LLMs may pass invalid values; validation happens at runtime.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 62 | 2026-07-28+ | v2 |
query_layer 'where' parameter accepts arbitrary SQL. No validation or injection guards documented. SSRF guard on URL is good, but SQL injection risk remains.