Loading ToolBench
A FastAPI backend with React frontend for air quality monitoring, forecasting, and comparative analysis with MCP tool integration for scraping, comparing cities, and forecasting PM2.5/PM10 levels.
Air Quality MCP server has clear, domain-specific tool names that follow verb_noun conventions (scrape_city, compare_cities, forecast_city, forecast_multi). All four tools have descriptions and input schemas. However, descriptions are quite brief (averaging ~120 chars), lacking WHEN/WHY context that LLMs need for tool selection. Parameter descriptions are present but minimal. Output schemas are not documented anywhere in the provided code. Error handling is not visible, no recovery guidance, no categorization of retryable vs fatal errors. No tool annotations (readOnlyHint/destructiveHint) despite clear risk distinctions (scrape_city is WRITE, others are READ_ONLY). Schema completeness is moderate: all params are typed with reasonable constraints (min/max for integers, array items typed), but no enum constraints where applicable. Composition is reasonable, tools are single-purpose and parameters chain naturally (cities output → compare_cities input), but no evidence of idempotency guarantees or pagination for list-like results.
Compute KPIs over the last N days per city (n_points, mean_pm25, min_pm25, max_pm25) and pick best/worst (lower is better).
Forecast next H days of PM2.5 for one city with SARIMAX; returns yhat + CI.
Forecast next H days for multiple cities and rank best/worst by mean predicted PM2.5.
Fetch & cache hourly PM2.5/PM10 for a city over the last N days using Open-Meteo; upserts into MySQL.
Output schemas not documented. Tool descriptions specify WHAT is returned (e.g., 'returns yhat + CI', 'returns KPIs') but the actual field names, types, and structure are not visible. LLMs cannot plan downstream tool calls without knowing the response structure.
Descriptions are too brief (averaging 120 chars). Current descriptions answer WHAT but lack WHEN/WHY context. E.g., 'forecast_city' description doesn't explain when to use single-city vs multi-city forecasting, or what SARIMAX implies about data requirements.
No tool annotations despite clear risk differences. scrape_city is marked WRITE (modifies MySQL) but forecast_* are READ_ONLY. MCP spec 2026-07-28 supports destructiveHint, readOnlyHint, and idempotentHint annotations in tool definitions, none are visible in the schemas provided.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 58 | <=2025-11-25 | v2 |
| 2026-03-09 | D | 59 | - | v1 |
No error handling documentation. If scrape_city fails due to invalid city name or API timeout, the description doesn't tell the LLM what to do next (retry, ask user, call a discovery tool). No recovery guidance.
Parameter descriptions lack detail on constraints and expected formats. E.g., 'days' param has min/max (1 - 90) in schema but description doesn't explain what happens at boundaries or why the limit exists. 'city' parameter has no hint about valid formats (e.g., must match Open-Meteo city names).
No idempotency guarantees documented. scrape_city 'upserts' into MySQL (implying idempotency), but this is not stated in the description. LLMs need explicit idempotency guarantees to safely retry on transient failures.
No pagination guidance for list-like results. compare_cities and forecast_multi accept arrays of cities but do not document result limits. If an LLM passes 50 cities, will the API time out, or return all results in a bloated response?