Loading ToolBench
City of New Orleans GIS — New Orleans, Louisiana open geospatial data (ArcGIS). Provides access to parcels, zoning, public safety & city services datasets via ArcGIS Feature Services.
Three well-named, read-only tools with clear descriptions and complete input schemas. All tools follow verb_noun naming (search_, query_, layer_info). Descriptions are 150 - 250 chars, within the 10 - 1024 baseline. All parameters have type definitions and descriptions. However, output schemas are not documented, LLMs cannot predict response structure for downstream chaining. No error handling guidance (e.g., 'if query fails, try a simpler where clause'). Missing tool annotations (readOnlyHint). Parameter constraints are present but could be more explicit (e.g., limit ranges stated in description but not as JSON Schema min/max).
Get an ArcGIS Feature/Map Service layer's schema by url: fields (name + type), geometry type, total record count, and capabilities.
Query an ArcGIS Feature Service / Map Service layer by its url (from search_datasets). SQL-like `where`, comma-separated `out_fields`, `order_by`, `limit`, `offset`. Returns attribute rows (and geometry). Use where="1=1" + out_fields="*" to sample.
Search City of New Orleans GIS open geospatial datasets (parcels, zoning, public safety & city services) by keyword. Returns each dataset's name, summary, record_count, owner/org, and its Feature Service `url` — pass that url to query_layer / layer_info.
Output schemas not documented. LLMs cannot predict response structure (e.g., does search_datasets return an array or object? What fields are in each dataset?). This breaks tool chaining and forces agents to guess.
No error handling guidance. If query_layer fails (e.g., invalid SQL where clause), the description does not tell the LLM what to do next. Should include: 'If query fails, verify the where clause syntax or try where="1=1" to sample all records.'
Parameter constraints (limit 1 - 50, limit 1 - 2000) stated in descriptions but not formalized in JSON Schema (no min/max properties). LLMs cannot parse natural-language constraints reliably.
Missing tool annotations. All three tools are read-only and idempotent, but inputSchema lacks readOnlyHint and idempotentHint annotations. Agents cannot infer safety properties without explicit hints.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 62 | 2026-07-28+ | v2 |
No pagination guidance in search_datasets description. If a query returns 20 datasets but totalCount is 500, the LLM has no way to know pagination is available. Should state: 'Use limit to paginate; response includes total count.'