Loading ToolBench
City of Cupertino GIS — Cupertino, California open geospatial data (ArcGIS). Provides access to parcels, zoning, public works & city services datasets via ArcGIS Feature Services.
Three well-named, read-only tools with clear descriptions and complete input schemas. Naming follows verb_noun pattern (search_datasets, query_layer, layer_info). Descriptions are substantive (150 - 200 chars) and explain what each tool does and when to use it. All parameters have type definitions and descriptions. However, output schemas are not documented, LLMs cannot predict response structure. Parameter constraints (e.g., limit 1 - 50, 1 - 2000) are stated in descriptions but not enforced via JSON Schema enums/min/max. No error handling guidance or recovery paths documented. Missing required field declarations in some schemas (e.g., search_datasets has no 'required' array). No tool annotations (readOnlyHint, idempotentHint) despite all being read-only and idempotent.
Get an ArcGIS Feature/Map Service layer's schema by url: fields (name + type), geometry type, total record count, and capabilities.
Query an ArcGIS Feature Service / Map Service layer by its url (from search_datasets). SQL-like `where`, comma-separated `out_fields`, `order_by`, `limit`, `offset`. Returns attribute rows (and geometry). Use where="1=1" + out_fields="*" to sample.
Search City of Cupertino GIS open geospatial datasets (parcels, zoning, public works & city services) by keyword. Returns each dataset's name, summary, record_count, owner/org, and its Feature Service `url` — pass that url to query_layer / layer_info.
Output schemas not documented. LLMs cannot predict response structure (fields, types, nesting). Forces agents to guess or fail on unexpected formats.
Parameter constraints (limit ranges, enum values) stated only in descriptions, not in JSON Schema. LLMs cannot parse description text for constraints, they need formal min/max/enum declarations.
No error handling guidance. Tools throw generic errors ('ArcGIS Hub: 404', 'Invalid input') without recovery hints. LLMs cannot determine if error is retryable, user-fixable, or fatal.
Missing 'required' field in search_datasets inputSchema. Unclear which parameters are mandatory vs optional. query_layer and layer_info correctly declare 'required: ["url"]'.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 60 | 2026-07-28+ | v2 |
No tool annotations (readOnlyHint, idempotentHint). All three tools are read-only and idempotent, annotations would help agents reason about safety and retry logic.