Loading ToolBench
MCP server for integrating with Report Portal to manage test launches, investigate failing tests, and mark tests as flaky
Server has 3 tools with basic schema and description coverage, but significant quality gaps. Two tools (get_launches, get_test_items_to_investigate) have adequate input schemas with enums and descriptions. However, get_test_items_to_investigate has a critical naming issue ('_to_investigate' is vague and combines multiple concerns). The mark_tests_flaky tool uses an array parameter that lacks validation constraints. All tools lack output schema documentation, LLMs cannot know what fields to expect in responses. Error handling is absent: tools do not guide recovery on failures. No tool has per-parameter descriptions explaining format/range constraints. Tool descriptions are generic and under-optimized for LLM selection (e.g., 'Fetch recent test items marked for investigation' does not explain WHEN to use this tool vs alternatives, or what the response structure is). The server provides prompts but no resources, logging, or error reporting. Transport is HTTP (Streamable), which is current, but stateless request handling and per-request _meta logLevel are not evident.
Fetch recent launches from Report Portal
Fetch recent test items marked for investigation from ReportPortal
Mark tests that passed as flaky in ReportPortal
Tool name 'get_test_items_to_investigate' is ambiguous. '_to_investigate' is not an action verb and conflates discovery (what items exist) with filtering (which ones need investigation). Violates verb_noun naming convention. LLM cannot infer the precise action from the name alone.
No output schema documented for any tool. LLMs do not know what fields to expect in responses. get_launches returns {launches}, get_test_items_to_investigate returns {items, totalRecords}, mark_tests_flaky returns {success, updatedCount}, but these are inferred from code, not declared. Tools return both 'text' and 'structuredContent', but the structured output format is not specified in tool metadata.
Tool descriptions lack context for LLM selection. 'Fetch recent launches from Report Portal' does not explain WHEN to use this tool, what prerequisites exist, or what the response structure is. Descriptions are 50 - 65 characters but should be 80 - 200 for LLM comprehension. No mention of pagination (limit parameter exists, but no mention of result limits or offset/cursor handling).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 47 | 2025-06-18+ | v2 |
| 2026-03-09 | F | 43 | - | v1 |
Parameter 'testItemIds' in mark_tests_flaky lacks validation constraints and description clarity. Schema declares type='array', items.type='number', minItems=1, but no maximum array size, no guidance on valid ID ranges, no statement of what happens if an ID is invalid (partial failure? total failure?). Description says only 'Array of test item IDs to mark as flaky' but does not explain format or error handling.
No error recovery guidance. If a tool fails (e.g., 'Test item not found', 'API timeout', 'Unauthorized'), the LLM has no hint about next steps. Error responses should indicate: retryable? user-fixable? fatal? Current catch-all error handling does not categorize or advise.
mark_tests_flaky is a destructive tool (writes state) but has no confirmation step, dry-run option, or destructiveHint annotation. If an agent accidentally passes incorrect IDs, tests will be marked flaky without recourse. No tool annotation metadata detected in any tool.
Parameter descriptions are minimal or absent. 'limit' in get_launches says 'Number of launches to fetch' but does not explain what 'recent' means (last N days? last N runs? sorted by timestamp?). 'team' enum in get_test_items_to_investigate provides valid values but no explanation of what team mapping means in the ReportPortal domain.