Loading ToolBench
Eval harness measuring the tool-surface design claims from the 'good MCP tool surface' essay
This is an evaluation harness, not a production MCP server. Tool definitions are present with schemas and descriptions, but quality is inconsistent. 35 tools span two domains (Census demographics, SEC filings). Many tools exhibit fragmentation anti-pattern: 12 tools (list_annual_reports, list_quarterly_reports, list_current_reports, list_registration_statements, list_proxy_statements, get_recent_reg_cf_offerings, get_recent_reg_d_offerings, get_recent_reg_a_offerings, get_company_revenue, get_company_net_income, get_company_total_assets, get_company_eps, get_company_cash, get_company_liabilities, get_insider_buys, get_insider_sells, search_filings_all, search_filings_by_company, search_filings_in_daterange) are specialized wrappers around parameterized tools (list_filings, get_company_facts, get_insider_trades, search_filings), violating the single-responsibility principle. Descriptions are terse (avg ~60 chars, well below the 194-char baseline for A+ tools). Parameter descriptions are minimal. No output schemas documented. Error handling absent. The submit_answer tool is a meta-tool for the evaluation harness, not a domain tool.
Compare a metric across multiple ZIP codes, sorted by value
Get a specific ACS variable value for a ZIP code
Get a public company's reported cash and equivalents from its XBRL facts.
Get a public company's reported earnings per share from its XBRL facts.
Get a public company's reported financial metric from its XBRL facts
Get a public company's reported total liabilities from its XBRL facts.
Get a public company's reported net income from its XBRL facts.
Severe tool fragmentation: 12 specialized wrappers (list_annual_reports, get_company_revenue, get_insider_buys, search_filings_all, etc.) duplicate parameterized tools (list_filings, get_company_facts, get_insider_trades, search_filings). LLMs waste reasoning cycles choosing between near-identical tools.
Descriptions are terse (avg ~50 chars, well below 194-char baseline). Most lack context for when/why to call the tool. E.g., 'Get income statistics for a US ZIP code from Census data' does not explain prerequisites, return structure, or when to use vs. get_acs_variable.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 46 | 2026-07-28+ | v2 |
Get a public company's reported revenue from its XBRL facts.
Get a public company's reported total assets from its XBRL facts.
Get demographic statistics for a US ZIP code
Get education statistics for a US ZIP code
Get housing statistics for a US ZIP code from Census data
Get income statistics for a US ZIP code from Census data
Recent insider purchase transactions (Section 16) for a company, newest first.
Recent insider sale transactions (Section 16) for a company, newest first.
Recent insider transactions (Section 16) for a company, newest first
List recent offerings by regulation type (Reg CF, Reg D, or Reg A+), newest first
List recent Reg A+ offerings (Form 1-A), newest first.
List recent Reg CF crowdfunding offerings (Form C), newest first.
List recent Reg D private offerings (Form D), newest first.
List an issuer's most recent annual reports (Form 10-K), newest first.
List an issuer's most recent current reports (Form 8-K), newest first.
List an issuer's filings of a specific form type, newest first
List an issuer's most recent proxy statements (DEF 14A), newest first.
List an issuer's most recent quarterly reports (Form 10-Q), newest first.
List an issuer's most recent registration statements (Form S-1), newest first.
Resolve a stock ticker symbol or company name to its SEC CIK and identity
Resolve a company name (e.g. 'Apple Inc.') to its SEC CIK and identity.
Resolve a stock ticker symbol (e.g. 'AAPL') to its SEC CIK and identity.
Look up basic demographic data for a US ZIP code
Full-text search across SEC filing documents
Full-text search across all SEC filing documents.
Full-text search across SEC filings, scoped to one company.
Full-text search across SEC filings filed within a date range.
Call this once you can answer the user, with your final answer.
No output schemas documented. LLMs cannot plan downstream tool calls or extract required fields (e.g., does get_income return a dict with 'median_household_income' key?). Breaks tool chaining.
No error handling guidance. Tools return raw API errors or failures with no recovery hints. E.g., if a ZIP code is invalid, does the tool return a 404 or a structured error? How should the LLM respond?
Parameter descriptions are minimal or absent. E.g., 'metric' in compare_zips lacks guidance on valid values (median_household_income? population? housing_units?). LLMs must guess or hallucinate.