Loading ToolBench
Cloud-wired intelligent testing context server for test creation, auto-healing, coverage gaps, and test impact analysis. Provides tools for test management, coverage reporting, failure diagnosis, and AI-powered test generation.
TestRelic MCP has 31 tools with consistent naming (verb_noun pattern: tr_get_*, tr_list_*, tr_create_*, tr_diagnose_*, etc.). All tools have descriptions (avg ~80 chars, within baseline 34-392 range). Input schemas are present and typed for all tools. However, descriptions are often terse (e.g., 'Digest read of repository memory', 'Check the health status of the MCP server and upstream services') and lack actionable context for LLM selection. Parameter descriptions are minimal, most are 1-2 sentences without format constraints, ranges, or dependency hints. Output schemas are not documented in the provided source. Error handling guidance is absent. No tool annotations (readOnlyHint/destructiveHint) despite clear risk classifications (READ_ONLY vs WRITE). Tool composition is sound (single responsibility), but descriptions don't explain WHEN to use each tool vs. similar ones (e.g., tr_diagnose_run vs tr_ai_rca vs tr_heal_run).
Execute an AI tool with specified input
AI-powered root cause analysis of a failing run
Analyze a code diff for test impact
List connected applications for the organization
Export an artifact in a specified format
Ask the platform AI a question with optional context
Identify coverage gaps and uncovered journeys
Generate a coverage report for a repository
Output schemas not documented. LLMs cannot plan downstream tool calls or extract required fields (e.g., what does tr_diagnose_run return? Does it include run_id for chaining to tr_create_jira?). Baseline: 100% of A+ tools document return types.
Descriptions lack actionable context. Most are 1-2 sentences without explaining WHEN to use the tool vs. similar ones. E.g., tr_diagnose_run, tr_ai_rca, and tr_heal_run all operate on failing runs but descriptions don't clarify the distinction. Baseline: descriptions should be 50-200 chars and answer WHAT, WHEN, and WHAT IT RETURNS.
Inferred effective spec: 2025-06-18+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 52 | 2025-06-18+ | v2 |
Create a Jira ticket from test run data
Get detailed information about a repository
Diagnose a failing test run and collect failure signals
Dismiss a flaky test flag
Run a type check and dry-run on generated test code
Fetch a cached result by cache key
Audit flaky tests in a repository
Generate test code from a test plan
Digest read of repository memory
Propose a healing patch for a failing test run
Check the health status of the MCP server and upstream services
Filterable list of repository memories with category, search, status, and limit filters
List repositories accessible to the authenticated user
Connect a marketplace application with validated credentials
Get details about a specific marketplace application
List available marketplace applications
Start OAuth flow for a marketplace application
Validate credentials for a marketplace application
Plan a test for a coverage gap or journey
Fetch recent test runs for a repository
Write a repository memory entry (requires mcp:memory PAT scope)
Select tests to run based on diff impact analysis
Correlate test run impact to user-facing journeys
No tool annotations despite clear risk classifications. Tools marked WRITE (tr_save_repo_memory, tr_create_jira, tr_generate_test, tr_marketplace_connect, tr_dismiss_flaky) lack destructiveHint. READ_ONLY tools lack readOnlyHint. Agents cannot distinguish safe from risky operations.
Parameter descriptions are minimal. Most lack format constraints, ranges, or dependency hints. E.g., 'limit' params have no min/max (baseline: 1-100 or 1-365). 'status' param in tr_list_repo_memories lacks enum values. LLMs cannot validate inputs.
No error handling guidance. Tools lack recovery hints (e.g., 'If run_id not found, call tr_recent_runs() first'). No categorization of errors as retryable vs. user-fixable. Agents cannot self-correct on failures.