Loading ToolBench
Analyzes HTML test reports (Pitest mutation and JaCoCo branch coverage) for Java projects
This MCP server has three tools for analyzing Java test reports (mutation and branch coverage). All three tools have: (1) generic naming that doesn't follow verb_noun conventions clearly; (2) descriptions that are present but lack clarity on what the LLM should do with the output; (3) input schemas with minimal type information and no parameter descriptions beyond a single line; (4) no documented output schemas; (5) no error guidance that tells the LLM how to recover from failures. The code shows hard-coded Windows paths (backslashes), platform-specific assumptions (os.name checks), and fragile HTML/XML parsing that will fail silently on malformed reports. Parameter descriptions are present but generic and do not explain valid ranges, constraints, or expected formats.
Show test report for branch test Returns: A string containing the analysis results, including branch coverage,total number of branches and at which line the branch is.
Show test report for mutation test Returns: A string containing the analysis results, including mutation coverage,total number of mutations and at which line what mutators were applied in missed mutations.
Show test report overview Returns: A string containing the analysis results, including mutation coverage and branch coverage (If you want to see the detailed results, please use the "getBranchTestReport" or "getMutationTestReport" tool)
Tool names use camelCase getter convention (getMutationTestReport) rather than verb_noun_snake_case (get_mutation_test_report). Naming is not idiomatic MCP and makes tool selection harder for LLMs that expect action verbs at the start.
Input schemas present but severely under-specified. All three tools only define 'path' and optionally 'className' as strings with minimal descriptions. No type information beyond 'string'; no constraints (length, format, regex); no indication of valid ranges or allowed values.
No documented output schemas. Tools return free-text strings with embedded coverage percentages and line numbers. LLMs cannot parse this unstructured output reliably, they must use regex or string splitting, which is fragile and wastes tokens. Descriptions say 'Returns: A string containing...' but never define the string format.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 39 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 27 | - | v1 |
Error handling returns bare strings ('No pitest report found', 'Error when parsing pitest report'). No guidance on what the LLM should do next (retry? ask user? call another tool?).
Tool descriptions lack clarity on WHEN to use each tool versus alternatives. For example, getTestReportOverview says 'If you want to see detailed results, please use... [other tools]', but under what circumstances should the agent choose one over the other? This forces the LLM to guess.
Hard-coded Windows paths (backslashes in r'\target\pit-reports\...'). Code also checks os.name but inconsistently, Windows path separators appear throughout. A project on Unix/Linux will fail silently when trying to find report files. Parameters accept 'path' but do not document whether it should be Windows, Unix, or platform-agnostic.
Parameter descriptions do not state what format 'path' should take. Is it absolute? Relative? Can it be a Path object or must it be a string? No mention of validation. 'className' is described as 'The name of the class (without package)' but doesn't specify whether it expects 'MyClass', 'my_class', 'MyClass.java', etc.
No idempotence guarantees. If an LLM retries a tool call (due to transient parse error), could the tool produce different results or side effects? Descriptions do not clarify. Tools read from disk without locking, so concurrent calls might race.