Loading ToolBench
MCP server for running pytest tests with marker filtering and argument customization
Single tool 'run_pytest' has a clear verb-noun name and acceptable description (157 chars), but critical gaps undermine quality. Input schema is present with three parameters, all with descriptions and types. However, the tool lacks output schema documentation, the return dict structure is undocumented, forcing LLMs to infer field meanings. No error handling guidance (e.g., what to do if pytest is not installed, or if test_path doesn't exist). No parameter constraints (e.g., max length for extra_args to prevent command injection). The description includes example values ('smoke', 'regression and not slow') which LLMs may reuse literally. No mention of idempotency or side effects. Baseline: 194 chars avg description (this is 157, acceptable); 100% of A+ tools document return types (this does not).
Run pytest with optional marker filtering and extra arguments. Args: marker - An expression for pytest -m, for example: "smoke" or "regression and not slow". test_path - The path (relative to the project root) from which to run tests. Default: "test_cases". extra_args - Additional flags for pytest, for example: "-vv --maxfail=1".
Output schema not documented. Return dict has keys 'command', 'exit_code', 'stdout', 'stderr' but LLM cannot infer structure, types, or meaning without explicit schema.
No error handling guidance. Tool does not document what happens if pytest is missing, test_path is invalid, or extra_args contains malicious flags. No recovery hints for LLM.
Example values in description ('smoke', 'regression and not slow') risk literal reuse by LLM. Should use enum constraints or formal syntax instead.
No input validation or constraints. 'extra_args' parameter accepts arbitrary strings; no length limit, no injection prevention, no validation of pytest flags.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 54 | <=2025-11-25 | v2 |
No idempotency or side-effect declaration. Tool modifies test state (runs tests, may create reports) but description does not warn LLM about retry implications.