MCP server exposing an mcp-swe-gym workspace to a coding agent
Server defines 8 tools with clear names (verb_noun pattern: task_info, list_files, read_file, search_code, write_file, run_tests, get_diff, submit). All tools have descriptions (10-200 chars, within baseline 34-392 range). Input schemas present for all tools using Zod validation. However, schemas lack explicit type declarations in the visible schema objects (e.g., task_info has empty {}), and parameter descriptions are minimal. Output schemas are not documented. Error handling uses guard() wrapper but lacks recovery guidance or categorization. No tool annotations (readOnlyHint/destructiveHint) despite clear risk levels (READ_ONLY vs WRITE). Composition is sound, tools chain logically (list_files → read_file → write_file → run_tests → submit). Security: write_file refuses protected paths; no secrets exposed as parameters.
The unified diff of every change made in the workspace so far.
List files and directories within the workspace.
Read a file, optionally limited to a line range (1-indexed, inclusive).
Run the task test suite and return the per-test outcome. Use this to check your work before submitting.
Regular-expression search across the workspace. Returns file, line number and the matching line.
Grade the current state of the workspace and return the final report. Reward is zero if any protected file was modified.
The bug report to resolve, plus the language and how the test suite is run. Read this first.
task_info, run_tests, get_diff, submit have empty or minimal input schemas ({}). LLMs cannot infer parameter constraints or types.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract structured data. Baseline: 100% of A+ tools document return types.
No tool annotations (readOnlyHint/destructiveHint/idempotentHint) despite clear risk levels. write_file is destructive; read tools are idempotent. Missing current spec patterns.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 55 | 2026-07-28+ | v2 |
Write a file in the workspace, creating it if needed. Refuses protected paths such as the test suite.
Error handling via guard() wrapper returns generic 'Error: <message>' without recovery guidance or error classification. Baseline: errors must tell LLM what to do next.
Parameter descriptions for list_files (path, recursive), read_file (start_line, end_line), search_code (glob, max_results) are present but minimal. Baseline: 72 chars average. No format constraints or examples.