Loading ToolBench
MCP (Model Context Protocol) server for TestPicker.jl. Exposes Julia testing functionality via MCP, enabling LLMs to discover, run, and inspect tests programmatically.
TestPickerMCPServer demonstrates solid tool design with clear naming conventions (verb_noun pattern: list_*, run_*, get_*, activate_*) and comprehensive descriptions (avg 180 chars, well within 10-1024 range). All 8 tools have explicit MCPTool registrations with descriptions and parameter schemas. However, critical gaps exist: (1) No output schemas documented anywhere, LLMs cannot predict return structure; (2) Parameter descriptions lack format/constraint details (e.g., 'query' params don't specify fuzzy-match semantics); (3) No error handling guidance, tools don't explain recovery paths; (4) Missing tool annotations (readOnlyHint/destructiveHint) despite clear risk classifications in metadata. Tool composition is sound (single responsibility, clear chains: list_testfiles → run_testfiles → get_testresults), but lack of output documentation and error guidance prevents A-grade scoring.
Activate a different Julia package directory and update the server's active package. Use this to switch between different packages without restarting the server.
Get the status of the currently active package. Returns package name, UUID, path, and version if a package is active. Returns a 'no_package' status with instructions if no package is currently active.
Retrieve detailed failures and errors from the most recent test run. Returns arrays of failures (assertion failures) and errors (exceptions), each with test expression, file location, error message, and context. Returns empty arrays if all tests passed.
List all @testset blocks found in test files. Returns label, file path, line start, and line end for each test block. Useful for discovering available testsets before running specific ones.
List all test files in the current Julia package. Returns test directory path, list of test files (relative paths), and count. Optionally filter files with a fuzzy query string.
No output schemas documented. LLMs cannot predict return structure (fields, types, nesting). Tools return 'status', 'summary', 'failures', 'errors' but no schema visible in code or docs.
Missing tool annotations. Risk metadata (READ_ONLY, WRITE) exists in comments but not in MCPTool definitions. No readOnlyHint or destructiveHint fields visible.
Parameter descriptions lack format/constraint details. 'query' params describe fuzzy matching but don't specify case sensitivity, max length, or matching algorithm. 'pkg_dir' doesn't validate path format or existence.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 63 | 2026-07-28+ | v2 |
Run all tests in the package test suite. Returns status, number of files run, summary of results, and whether there are failures. Use get_testresults to see detailed failure information.
Run specific @testset blocks matched by fuzzy queries. First filters test files (optional), then filters testsets within those files (required). Useful for running a specific testset without running the entire file.
Run specific test file(s) matched by a fuzzy query string. The query is matched against test file paths (e.g., 'feature' runs test_feature.jl). Returns list of files run and execution status.
No error handling guidance. Tools don't document failure modes (e.g., 'no tests found', 'invalid package path', 'test execution timeout') or recovery steps for LLMs.
No pagination support documented. list_testfiles and list_testblocks could return hundreds of items; no limit, offset, or cursor parameters visible.