Loading ToolBench
An MCP server that puts Socrata open data on tap for LLM clients
Four well-named, read-only tools with clear descriptions and complete input schemas. Tool names follow verb_noun pattern (search_datasets, get_schema, query_dataset, profile_column). All tools have descriptions 100+ chars explaining what they do and when to use them. Input schemas include types and descriptions for all parameters. However, output schemas are not documented in the source code, the server returns 'list[dict[str, Any]]' or 'dict[str, Any]' with no field-level documentation. This forces LLMs to infer structure from examples. Parameter descriptions lack format constraints (e.g., dataset_id format '4x4' is mentioned in descriptions but not formalized). Error handling is minimal, no recovery guidance or actionable error messages visible in the tool definitions.
Get a dataset's name, description, and column names/types. Call this before querying so column names in SoQL clauses are exact.
Show the most frequent values of one column with counts — a quick way to learn a column's vocabulary before writing a where clause.
Run a SoQL query against a dataset. Clauses mirror SQL: e.g. select='borough, count(*) as n', where="created_date > '2026-01-01'", group='borough', order='n DESC'. Rows are capped at 1000 per call.
Search a Socrata portal's catalog by keyword. Returns dataset ids, names, and descriptions. Use the returned id with get_schema/query_dataset.
Output schemas not documented. Tools return 'list[dict[str, Any]]' or 'dict[str, Any]' with no field-level documentation. LLMs cannot plan downstream calls or extract specific fields without inferring structure from examples.
Parameter format constraints not formalized. dataset_id format ('4x4 form, e.g. erm2-nwe9') is mentioned in descriptions but not declared as a regex pattern or enum. LLMs may pass invalid IDs.
No error recovery guidance. Tool definitions do not indicate what errors are possible, whether they are retryable, or what the LLM should do next (e.g., 'If dataset not found, try search_datasets first').
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | 2025-06-18+ | v2 |
Result limits not enforced in tool descriptions. query_dataset mentions '1000 per call' cap in description but does not declare it as a max constraint on the limit parameter. profile_column's top parameter lacks min/max bounds.
SoQL syntax errors not handled gracefully. query_dataset accepts free-form SoQL strings (select, where, group, order) with no validation. Invalid syntax will fail at the API level with no guidance for the LLM to self-correct.