NYC Open Data — US government open data (data.cityofnewyork.us) via the Socrata SoQL API: NYC agencies — 311, transportation, housing, permits, crime & health. Keyless.
Three tools with complete schemas and descriptions, but significant gaps in parameter documentation and output schema clarity. Tool names lack action verbs (datasets, query, metadata are nouns). Descriptions are adequate (100-200 chars) but lack WHEN/WHY guidance. Parameter descriptions exist but omit constraints, ranges, and format details. No output schema documented. Error handling is minimal. Composition is sound, tools chain logically (datasets → metadata/query). Security is acceptable (read-only, no secrets exposed). Naming could be clearer: 'search_datasets', 'run_query', 'get_metadata' would better signal intent to LLMs.
Search the NYC Open Data catalog of open datasets by keyword. Returns each dataset's resource_id, name, description, category and update date — pass the resource_id to query/metadata.
Get a NYC Open Data dataset's schema + metadata (columns, types, row count, category, last-updated) by resource_id, e.g. "8wbx-tsch".
Run a Socrata SoQL query against a NYC Open Data dataset by resource_id (e.g. "8wbx-tsch"). Filter with where/select/group/order (SoQL clauses, without the leading $) plus limit/offset. Returns matching rows as JSON.
Tool names are nouns, not action verbs. 'datasets', 'query', 'metadata' lack clarity on intent. LLMs infer action from name alone, 'search_datasets', 'run_query', 'get_metadata' would be clearer.
Output schemas are not documented. Callers cannot predict response structure (e.g., does 'datasets' return array or object? What fields?). LLMs must guess, risking misuse.
Parameter descriptions lack constraints and formats. 'limit' says '1-100, default 20' but no type validation shown. 'where/select/group/order' accept free-form SoQL strings with no format guidance or examples of valid syntax.
No error handling guidance. If a resource_id is invalid or SoQL query fails, the tool throws a generic error. No recovery hints (e.g., 'Try search_datasets() first to find valid IDs').
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | 2026-07-28+ | v2 |
No pagination documentation in tool descriptions. 'datasets' accepts limit/offset but description doesn't explain how to iterate through large result sets or what total count is returned.