Loading ToolBench
Loudoun County GIS — Loudoun County, Virginia open geospatial data (ArcGIS). Provides access to parcels, addresses, zoning & public works datasets via ArcGIS Feature Services.
Three well-named, read-only tools with clear descriptions and complete input schemas. Naming follows verb_noun pattern (search_datasets, query_layer, layer_info). Descriptions are 150 - 250 chars, explaining WHAT and WHEN. All parameters have type and description. However, output schemas are not documented, LLMs cannot predict response structure for downstream planning. Parameter constraints (e.g., limit 1 - 50) are stated in descriptions but not enforced as JSON Schema enums/min/max. No error handling guidance; raw HTTP errors will confuse agents. Missing idempotent/readOnly annotations.
Get an ArcGIS Feature/Map Service layer's schema by url: fields (name + type), geometry type, total record count, and capabilities.
Query an ArcGIS Feature Service / Map Service layer by its url (from search_datasets). SQL-like `where`, comma-separated `out_fields`, `order_by`, `limit`, `offset`. Returns attribute rows (and geometry). Use where="1=1" + out_fields="*" to sample.
Search Loudoun County GIS open geospatial datasets (parcels, addresses, zoning & public works) by keyword. Returns each dataset's name, summary, record_count, owner/org, and its Feature Service `url` — pass that url to query_layer / layer_info.
Output schemas not documented. LLMs cannot predict response structure (fields, types, nesting) for downstream tool calls or data extraction. search_datasets returns {total, count, datasets[{id, name, summary, org, org_id, record_count, type, url}]} but this is invisible to the agent.
Parameter constraints (limit 1 - 50, offset, order_by) stated only in descriptions, not in JSON Schema. LLMs cannot parse natural-language constraints reliably. Should use minValue, maxValue, enum, or pattern in inputSchema properties.
No error handling guidance. Raw HTTP errors (e.g., 'ArcGIS query: 404') give agents no recovery path. Should return structured errors with actionable next steps: 'Layer not found. Verify url ends in /FeatureServer/<n> or /MapServer/<n>.'
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 60 | 2026-07-28+ | v2 |
Missing tool annotations. All three tools are read-only and idempotent, but no readOnlyHint or idempotentHint declared. Agents cannot distinguish safe tools from destructive ones without explicit metadata.
query_layer accepts free-form SQL 'where' clause without validation. Agents could pass malicious or malformed SQL. Should validate syntax or document constraints (e.g., 'WHERE clause must be valid ArcGIS SQL; no DROP, DELETE, or INSERT allowed').