Loading ToolBench
Read-only MCP server over the GHCNh weather warehouse and WeatherKit forecasts. Provides 13 tools covering schema discovery, station discovery, historical observations, climate anomalies, WeatherKit forecasts, and open-ended SQL queries.
Strong foundation with 12 well-named, verb-prefixed tools covering a coherent domain (weather data discovery, observation retrieval, forecasting, and SQL access). All tools have descriptions (avg 180 chars, within 10-1024 baseline). Input schemas are complete with types and descriptions for all parameters. Output schemas are documented in descriptions. Key strengths: clear naming (list_*, get_*, summarize_*, describe_*), comprehensive parameter descriptions with constraints (enums for measures/granularity), and sensible defaults (limit clamping, date parsing). Gaps: no explicit error handling guidance in descriptions (e.g., what happens if station_id is invalid?), no output schema formally documented in tool definitions, and no tool annotations (readOnlyHint, idempotentHint). All tools are read-only, which is safe, but this is not declared in metadata.
Report what the warehouse actually holds right now. Row count, station count, earliest and latest observation, and how far behind the present the newest reading is. Worth calling before drawing conclusions from a recent-period query: the GHCNh feed is republished upstream and has regressed before, so "no data for last week" is often a source condition rather than an empty result. Set `by_year` for the same breakdown per year, which is how you spot a partially-loaded year.
Describe one table: its columns, types, and units. Call this before run_sql to understand what columns are available and what they measure. The descriptions include units and ranges where applicable.
Report what forecast data is available. Row count, station count, earliest and latest forecast time, and how far into the future the newest forecast extends.
Fetch hourly forecast for one station over a time range. Returns every measure at once (temperature, dew point, pressure, wind, visibility, precipitation), newest first. `start` is inclusive, `end` exclusive, both ISO dates.
Fetch raw hourly observations for one station over a date range. Returns every measure at once (temperature, dew point, pressure, wind, visibility, precipitation), newest first. `start` is inclusive, `end` exclusive, both ISO dates. This is the detail view — for anything spanning more than a few days, `summarize_observations` returns far less data for the same question. Wind, visibility and pressure are ~86-90% populated in recent data; nulls are genuine gaps in the source, not errors.
No tool annotations (readOnlyHint, idempotentHint) declared. All 12 tools are read-only and safe to retry, but this is not signaled in metadata. LLMs cannot infer safety properties and may avoid retrying on transient failures.
Error handling descriptions missing. Tools do not document what happens on invalid inputs (e.g., invalid station_id, malformed date, SQL injection attempt in run_sql). LLMs cannot self-correct without guidance.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | 2026-07-28+ | v2 |
Most recent forecast for one station, or for every station. Omit `station` for a roster-wide snapshot.
Most recent observation for one station, or for every station. Omit `station` for a roster-wide snapshot — that doubles as a freshness check, since a station stuck behind the others is visible immediately.
List all readable tables in the warehouse. Returns table names and their descriptions. Call this before describe_table to see what is available, or before run_sql to understand the schema.
List weather stations in the warehouse roster. Call this first when a question names a place rather than a station id — every other tool takes the 11-character `station` id this returns. The roster is 112 US stations. Filters combine with AND; omit all of them to list everything. `region` is one of MIDWEST, NORTHEAST, SOUTH, WEST. `name_contains` matches the station name case-insensitively, and also matches against the station id.
Execute a single read-only SELECT statement. For anything the typed tools do not cover, write a SELECT query. The query is validated before execution to ensure it is read-only and targets only approved tables. Call list_schema and describe_table first to understand the available tables and columns.
Aggregate one measure for one station into per-period min/avg/max. Similar to summarize_observations but for forecast data.
Aggregate one measure for one station into per-period min/avg/max. The workhorse for trend questions — "how did temperature move at O'Hare last summer" is a `granularity="day"` call, "how have summers trended" is `granularity="year"`. Returns newest period first.
Output schemas not formally documented in tool definitions. Descriptions mention what is returned (e.g., 'Returns every measure at once'), but no structured schema (field names, types) is visible in the tool metadata. Agents cannot plan downstream operations without inferring structure.
run_sql lacks explicit error guidance for SQL validation failures. Description mentions 'validated before execution' and references guard.py, but does not tell LLMs what error message to expect or how to recover (e.g., 'Query must be SELECT-only; UPDATE/DELETE rejected').
Parameter descriptions for date fields (start, end) do not specify timezone handling or whether times are UTC. ISO format is documented, but ambiguity around timezone could cause off-by-one errors in agent queries.