Loading ToolBench
City of Tulsa GIS — Tulsa, Oklahoma open geospatial data (ArcGIS). Provides access to parcels, zoning, public works & city services datasets via ArcGIS Feature Services.
Three well-named, read-only tools with complete input schemas and solid descriptions (avg 180 chars). All parameters are typed and described. However, output schemas are not documented, LLMs cannot predict response structure for downstream chaining. Error handling is minimal (generic HTTP status errors). No parameter constraints (enums, ranges) despite opportunities (e.g., limit 1-50 for search_datasets). Missing guidance on when to use each tool and how they compose. Security is good (SSRF guard on layerUrl, no secrets exposed), but no audit logging or permission gates.
Get an ArcGIS Feature/Map Service layer's schema by url: fields (name + type), geometry type, total record count, and capabilities.
Query an ArcGIS Feature Service / Map Service layer by its url (from search_datasets). SQL-like `where`, comma-separated `out_fields`, `order_by`, `limit`, `offset`. Returns attribute rows (and geometry). Use where="1=1" + out_fields="*" to sample.
Search City of Tulsa GIS open geospatial datasets (parcels, zoning, public works & city services) by keyword. Returns each dataset's name, summary, record_count, owner/org, and its Feature Service `url` — pass that url to query_layer / layer_info.
Output schemas not documented. LLMs cannot predict response structure (fields, types, pagination) for downstream tool calls or data extraction. search_datasets returns {total, count, datasets[]} but this is inferred from code, not declared in tool metadata.
No parameter constraints (enums, ranges, patterns). limit accepts 1-50 but is typed as bare 'number', LLMs may pass 0, 100, or -1. where clause is free-form string with no SQL injection guidance. offset lacks minimum bound documentation.
Error handling is generic. 'ArcGIS Hub: 404' or 'ArcGIS query: 500' gives LLM no recovery path. No distinction between retryable (timeout) vs user-fixable (invalid where clause) vs fatal (auth) errors. No guidance like 'Try search_datasets() first to find valid layer URLs.'
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 59 | 2026-07-28+ | v2 |
No tool composition guidance. Descriptions mention 'pass that url to query_layer / layer_info' but do not explain when to call each or what data flow looks like. LLMs must infer the workflow (search → query → info) from context.
query_layer description includes example SQL ('STATE = \'CA\' AND YEAR >= 2020') which LLMs may reuse literally in real calls. Replace with formal constraint documentation or remove example values.