Loading ToolBench
MCP Server for detecting API changes and generating test context
This server defines 6 tools with READ_ONLY risk profiles. All tools have descriptions and input schemas visible in src/index.ts. However, significant gaps exist: (1) Most parameter descriptions are present but generic; (2) Output schemas are NOT documented anywhere, no return type specifications visible; (3) Error handling is minimal, no recovery guidance or actionable error messages; (4) No idempotent/destructive hints despite all being read-only; (5) Parameter constraints are loose, no ranges, no enum validation for optional string parameters. The tools are well-scoped (one responsibility each) and names start with action verbs (get_, list_, analyze_), but descriptions lack depth on when to use each tool vs alternatives. Average tool score: 52/100. This is typical of community servers that prioritize basic functionality over LLM ergonomics.
Analyze an API file and extract all endpoint definitions with their methods, paths, and handlers
Get API changes between two branches or commits for a service. Detects new/modified API endpoints.
Get the full content of an API file for detailed analysis
Get pull requests for a service to find API changes in PRs
Get recent commits for a service to identify recent changes
List all configured services and their automation repo paths
No output/return schemas documented. Tools return data structures but LLMs cannot predict field names, types, or nesting. This breaks composition, an LLM cannot know what fields to extract after calling get_api_changes to pass to analyze_api_endpoint.
Optional parameters lack clear defaults or constraints. E.g., 'baseBranch' defaults to 'main' per code but description says 'default: main)', unclear if this is advisory or enforced. 'headBranch' has no default stated anywhere. LLMs cannot infer defaults from parameter names alone.
Error handling is implicit in code (try/catch returns 'Failed to fetch file: {error}') but no guidance in descriptions on how LLM should recover. No categorization (retryable vs fatal). Example: 'Service not found', should LLM call list_services first?
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 49 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 0 | - | v1 |
Parameter descriptions are functional but minimal. 'serviceName': 'Name of the service' is true but does not say: What happens if the service name is invalid? Should I call list_services first? What format is accepted (exact name, case-insensitive)? Descriptions should guide LLM decision-making.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) in the JSON schema. Although all tools are READ_ONLY per the risk profile, this is not signaled in the schema itself, LLMs must infer it from descriptions. Explicit annotations enable safer agent planning.
Pagination not explicitly supported. 'get_recent_commits' has a perPage param but no cursor or offset pattern. 'get_pull_requests' has no limit parameter at all. Tools returning lists should support pagination to avoid context window exhaustion. No documentation of result limits.
Tool descriptions lack WHEN/WHY guidance. 'Get API changes between two branches' describes WHAT, not WHEN to use it vs analyze_api_endpoint or get_recent_commits. Agents need help reasoning about tool selection when multiple tools operate on similar domains.