Loading ToolBench
MCP server for San Francisco's open data portal (DataSF)
Four well-named tools with clear, actionable descriptions (avg 156 chars). All tools have complete JSON Schema input definitions with type constraints, patterns, and bounds. Naming follows verb_noun convention (search_, list_, get_, query_). Schemas include validation (dataset_id pattern, query length limits, result limits 1-20). Error handling present via errorHandler.js with validation feedback. Main gaps: no output schemas documented, no parameter descriptions for optional fields (category, auto_correct), no error recovery guidance in descriptions, no tool annotations (readOnlyHint). Response structure inferred but not formally specified.
Get the schema (columns and data types) for a specific dataset. Call this before writing queries to learn the correct field names.
Browse available datasets from San Francisco's open data portal. Optionally filter by category. Returns recently updated or popular datasets.
Execute a SoQL (Socrata Query Language) query against a dataset. Supports auto-correction of column names. Returns query results as JSON.
Search for public datasets in San Francisco's open data portal by keywords. Returns dataset IDs, names, and descriptions.
Output schemas not documented. LLMs cannot plan downstream calls or extract fields without knowing response structure (e.g., what fields does search_datasf return? Is it {id, name, description} or {dataset_id, title, summary}?).
Optional parameters lack descriptions. 'category' in list_datasf and 'auto_correct' in query_datasf have no explanation of what they control or when to use them. Baseline: 100% of A+ tools describe all params.
No tool annotations (readOnlyHint). All four tools are read-only but lack explicit readOnlyHint in schema. Agents cannot distinguish safe tools from destructive ones without this metadata.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 59 | 2026-07-28+ | v2 |
Error messages lack recovery guidance. Validation errors return 'Invalid input: <message>' but do not suggest next steps (e.g., 'Invalid dataset_id format. Use xxxx-xxxx (e.g., 1234-abcd). Call list_datasf to find valid IDs.').
No pagination support documented. list_datasf and search_datasf cap results at 20 but do not expose offset/cursor or total_count. Large result sets may be truncated silently.