Loading ToolBench
A lightweight Model Context Protocol (MCP) server for Stata. Execute commands, inspect data, retrieve stored results (`r()`/`e()`), and view graphs in your chat interface. Built for economists who want to integrate LLM assistance into their Stata workflow.
mcp-stata presents a well-intentioned domain-specific toolkit with 10 tools covering Stata session management, data inspection, and execution. However, significant definition quality gaps prevent higher scoring. While all tools have descriptions and input schemas are present in toolkit_catalog_data.py, the descriptions are inconsistent in depth and many parameter descriptions lack specificity about constraints, formats, and error recovery. Schema definitions show types but lack comprehensive validation rules (enums for action parameters, numeric bounds, etc.). The tools follow a reasonable single-responsibility pattern (stata_run for execution, stata_inspect_data for inspection, stata_manage_graphs for graphs), but descriptions could be more prescriptive about when to use each tool and what fields to expect. Risk levels are declared (REVERSIBLE, WRITE, READ_ONLY) which is good for safety signaling, but error handling guidance is absent from descriptions. No evidence of output schema documentation or pagination support where applicable. Average parameter description length appears to be 15-25 characters, which is well below the 72-char baseline for A+ tools.
Break or cancel running work
Stata command documentation
Fetch r() / e() / s() stored results
Describe, codebook, summary, list, or get rows from the current dataset
Load datasets (sysuse / webuse / path)
List, export, or export_all graphs
Manage Stata sessions (create, stop, list, profile, UI channel, detect). This tool allows for orchestration of multiple Stata sessions, including their lifecycle management and configuration.
Parameter action values should be enums, not free-form strings. Tools stata_manage_session, stata_inspect_data, stata_manage_graphs, stata_read_log, and stata_control all accept action parameters with known fixed sets of values (create/stop/list/set_profile/etc.) but lack enum constraints. This allows LLMs to hallucinate invalid actions.
Output schemas are not documented. Tools like stata_run, stata_load_data, stata_get_results lack specification of what fields are returned, their types, and structure. LLMs cannot plan downstream tool calls or extract data without knowing the response shape.
Inferred effective spec: 2025-06-18+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 51 | 2025-06-18+ | v2 |
| 2026-03-09 | F | 41 | 1.0.0+ | v1 |
Tail or search log output
Execute Stata do-file code or ad-hoc commands
Poll background task progress
Descriptions are inconsistently detailed and often too brief. stata_run (50 chars), stata_load_data (40 chars), stata_manage_graphs (45 chars), and stata_get_help (27 chars) all fall below the 100-char minimum for LLM reasoning. Descriptions lack guidance on when to use each tool vs alternatives and what to expect.
Parameter descriptions lack specificity and validation hints. Many params have minimal or no guidance: 'code' in stata_run (what format?), 'filter' in stata_inspect_data (Stata syntax?), 'source' in stata_load_data (formats supported?), 'graph_name' in stata_manage_graphs (export format?). This forces LLMs to guess or causes tool calls to fail.
Error handling guidance is absent. Tools do not document what errors are possible, when to retry, or what to do next. E.g., if stata_load_data fails with 'file not found', should the LLM list available datasets? If stata_run fails with syntax error, should it call stata_get_help? Descriptions lack recovery pathways.
Tool purpose disambiguation is weak. The relationship between stata_manage_session (with 'detect' and 'profile' actions) and other tools is unclear. When should an LLM call stata_manage_session vs stata_run? When call stata_task_status vs stata_control? Descriptions don't explain the distinctions.
No pagination support declared for tools that may return large lists. stata_inspect_data 'get_rows' action, stata_get_results, and stata_read_log 'tail' action could return many items. No limit, offset, page_size, or next_cursor parameters documented.
Dual-purpose parameters create ambiguity. stata_read_log parameter 'query' serves two roles (line count for 'tail', search string for 'search'), LLMs may pass 10 expecting tail(10) but triggering search('10'). Split into 'lines' and 'search_term'.
Naming clarity gaps. 'stata_control' (too generic), 'stata_manage_graphs' (too generic, 'export_graph' would be clearer). 'stata_get_results' uses jargon (r/e/s) without explanation in the name itself.