Loading ToolBench
Search and fetch ~800K Federal Reserve economic time-series from the FRED API via MCP. STDIO or Streamable HTTP.
Strong tool definitions with complete schemas, clear descriptions, and proper parameter typing. All 7 tools have descriptions (avg 180 chars) and input schemas with typed parameters. Tool names follow verb_noun convention (fedreserve_get_*, fedreserve_browse_*, fedreserve_dataframe_*). However, output schemas are not documented in the provided source, parameter descriptions lack constraint details (ranges, formats), and error handling guidance is minimal. No evidence of per-tool risk annotations in the schema itself, though risk metadata is present in the evaluation context.
Navigate the FRED category tree. Returns the current category, its child categories, and — when viewing a leaf node with no children — a sample of series within it. Omit category_id to start at the root (ID 0).
List canvas dataframes materialized by fedreserve_get_observations, with provenance, TTL, column schema, and row count. Optionally filter to a single dataframe by name. Requires CANVAS_PROVIDER_TYPE=duckdb.
Drop a canvas dataframe by name. Idempotent — returns dropped: false when nothing matched. TTL handles cleanup in normal operation; this tool is for explicit early reclamation when an analysis session is complete. Requires CANVAS_PROVIDER_TYPE=duckdb and FRED_DATAFRAME_DROP_ENABLED=true.
Run a single-statement SELECT against canvas dataframes registered by fedreserve_get_observations. Standard DuckDB SQL — joins, aggregates, window functions, CTEs all supported. Reference dataframes by the df_<id> handles returned from fedreserve_get_observations or listed by fedreserve_dataframe_describe. Read-only: writes, DDL, DROP, COPY, PRAGMA, ATTACH, and external-file table functions are rejected. System catalogs are denied at the bridge layer. Optional register_as chains the result as a new dataframe with a fresh TTL. Requires CANVAS_PROVIDER_TYPE=duckdb.
Output schemas not documented. Tool descriptions state what is returned (e.g., 'Returns release name, link, scheduled dates, and a paginated list') but formal output schema definitions are not visible in source. LLMs cannot plan downstream tool calls without knowing response field names and types.
Parameter constraints lack specificity. Numeric parameters (category_id, series_limit, row_limit, preview) have no documented min/max bounds. Enum parameters (units, frequency, aggregation_method) are well-constrained, but free-form string params (series_ids, sql, name) lack format/length guidance. LLMs may pass invalid values without clear validation feedback.
Inferred effective spec: 2026-07-28+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 60 | 2026-07-28+ | v2 |
Fetch observation data (date + value pairs) for one or more FRED series over a date range. Supports FRED built-in unit transformations (lin, chg, ch1, pch, pc1, pca, cch, cca, log). Multi-series or >500-row results spill to a DataCanvas table — the response includes a `dataset` field with the handle to query via fedreserve_dataframe_query. Observation values are kept as strings to preserve trailing zeros.
Return metadata and associated series for a FRED release (e.g., Employment Situation, Consumer Price Index). Accepts a release_id integer or a release_search text query — release_search performs a substring match across all FRED releases; prefer release_id when you know the ID. Returns release name, link, scheduled dates, and a paginated list of its series. Provide exactly one of release_id or release_search.
Retrieve metadata for one or more FRED series — title, units, frequency, seasonal adjustment status, observation range, popularity, and notes. Accepts up to 50 series IDs. Returns partial success when some IDs fail.
Error handling guidance absent. No tool description explains what to do if a series_id is invalid, if a SQL query fails, or if a dataframe is not found. Descriptions state prerequisites (e.g., 'Requires CANVAS_PROVIDER_TYPE=duckdb') but do not guide recovery when those conditions are unmet.
Destructive tool (fedreserve_dataframe_drop) lacks confirmation or dry-run pattern. Description notes it is 'idempotent' but does not offer a preview or confirmation step before dropping a dataframe. Agents may accidentally delete analysis results.
Tool composition assumes agent knowledge of dataframe handles. fedreserve_get_observations returns a 'dataset' field with a handle (df_XXXXX_XXXXX), but the description does not explicitly state that this handle is the input to fedreserve_dataframe_query. Agents may not discover the chaining pattern without trial.