Loading ToolBench
Berks County GIS — Berks County, Pennsylvania open geospatial data (ArcGIS). Provides access to parcels, addresses, zoning & planning datasets via ArcGIS Feature Services.
Three well-named, read-only tools with clear descriptions and complete input schemas. All tools follow verb_noun naming (search_, query_, layer_info). Descriptions are substantive (150 - 250 chars) and explain purpose, prerequisites, and output. Input schemas include types and descriptions for all parameters. However, output schemas are not documented, LLMs cannot predict response structure. Parameter descriptions lack explicit constraints (e.g., 'limit: 1-50' is stated but not in schema as minItems/maxItems). Error handling is minimal, no recovery guidance or categorization. No tool annotations (readOnlyHint, idempotentHint) despite all being read-only and idempotent.
Get an ArcGIS Feature/Map Service layer's schema by url: fields (name + type), geometry type, total record count, and capabilities.
Query an ArcGIS Feature Service / Map Service layer by its url (from search_datasets). SQL-like `where`, comma-separated `out_fields`, `order_by`, `limit`, `offset`. Returns attribute rows (and geometry). Use where="1=1" + out_fields="*" to sample.
Search Berks County GIS open geospatial datasets (parcels, addresses, zoning & planning) by keyword. Returns each dataset's name, summary, record_count, owner/org, and its Feature Service `url` — pass that url to query_layer / layer_info.
Output schemas not documented. LLMs cannot predict response structure (fields, types, pagination). search_datasets returns {total, count, datasets[]} but this is inferred from code, not declared in tool definition.
No tool annotations. All three tools are read-only and idempotent, but lack readOnlyHint and idempotentHint. Agents cannot distinguish safe tools from destructive ones without explicit hints.
Parameter constraints not formalized in schema. 'limit: 1-50' and 'limit: 1-2000' are stated in descriptions but not enforced via JSON Schema minItems/maxItems. LLMs may pass out-of-range values.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | 2026-07-28+ | v2 |
Error handling lacks recovery guidance. callTool throws generic errors ('ArcGIS Hub: 404', 'Unknown tool') with no actionable next steps. Agents cannot self-correct or retry intelligently.
query_layer 'where' parameter accepts arbitrary SQL. No validation or injection guards documented. SSRF guard on URL is good, but SQL injection risk remains.