Loading ToolBench
A collection of Model Context Protocol servers for Git repository snapshots, COBOL parsing, JCL parsing, Mermaid diagram generation, and architecture guidance generation
Astra MCP is a multi-server aggregation with 15 tools spanning Git, COBOL/JCL parsing, Mermaid diagrams, and architecture guidance. While tools have basic descriptions and input schemas are generally present, quality is inconsistent. Many tool names lack clear action verbs (e.g., 'git.repo.snapshot.start' uses dot notation rather than verb_noun), descriptions are brief (average ~50 chars vs. production baseline of 194 chars), and parameter descriptions are sparse or missing entirely. Output schemas are not documented in any of the tool definitions visible. Error handling guidance is absent. No tool annotations (readOnlyHint, destructiveHint) despite clear risk distinctions (some tools are WRITE, others READ_ONLY). Composition is reasonable, paired start/status tools follow an async job pattern, but parameter reuse across tools is weak (job_id is minimal, no chaining context). This is mid-range community server quality: functional but below production standards.
Start COBOL Repository Parsing
Check COBOL Parsing Job Status
Start Data Pipeline Architecture Guidance Generation
Check Data Pipeline Architecture Guidance Generation Job
Generate Data Pipeline Architecture Guidance (blocking)
Generate Microservices Architecture Guidance (blocking)
Start Git Repo Snapshot
Tool naming convention is inconsistent: mix of verb_noun (e.g., 'git.repo.snapshot.start'), dot notation (domain.resource.action), and bare imperatives (generate_microservices_arch_guidance). LLMs infer intent from the verb, 'start' and 'generate' convey different semantics (async vs. blocking), but the naming scheme does not make this distinction clear across tools. Production baseline: 90% of A+ tools use simple verb_noun (e.g., start_git_snapshot, generate_arch_guidance).
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 50 | <=2025-11-25 | v2 |
| 2026-03-09 | D | 53 | - | v1 |
Check Snapshot Status
Start JCL Repository Parsing
Check JCL Parsing Job Status
Generate or sanitize Mermaid diagrams
Start Microservices Architecture Guidance Generation
Check Architecture Guidance Generation Job
Fetch and validate Raina AVC/FSS/PSS input JSON from a URL
Generate workspace documentation from artifacts
Descriptions are too brief and lack LLM-actionable guidance. Examples: 'Start Git Repo Snapshot' (28 chars, below minimum 34), 'Check Snapshot Status' (21 chars). Production baseline is 194 chars average for A+ tools. Descriptions do not answer: WHEN should the LLM call this instead of a similar tool? WHAT are the prerequisites? WHAT does it return? This forces LLMs to guess, leading to wrong tool selection, especially between blocking variants (generate_microservices_arch_guidance) and async variants (microservices.arch.guidance.start).
Parameter descriptions are missing or minimal. Examples: 'repo_url' has description 'The URL of the Git repository to clone', adequate. But 'workspace_id' across multiple tools (cobol.repo.parse.start, microservices.arch.guidance.start, data_pipeline.arch.guidance.start) is described only as 'Optional workspace identifier' with no guidance on what identifies a workspace, how to discover one, or what happens if omitted. LLMs cannot infer these details and will guess, leading to invalid calls. Production baseline: 100% of A+ tool params have descriptions; average 72 chars.
Output schemas are not documented for any tool. Tool definitions show input schemas (parameter types and descriptions) but no statement of what fields are returned, what data types, or what the response structure is. Production baseline: 100% of A+ tools have documented return types. Without output schemas, LLMs cannot plan downstream tool calls or know what fields to extract for chaining. Example: does git.repo.snapshot.start return {job_id, repo_path, estimated_time}? Or {job_id}? Unclear.
Tool annotation hints are completely absent. 15 tools include WRITE risk operations (e.g., git.repo.snapshot.start clones a repository, modifying filesystem; cobol.repo.parse.start and data_pipeline.arch.guidance.start trigger background jobs) and READ_ONLY operations (e.g., git.repo.snapshot.status, mermaid.diagram.generate). No tool declares readOnlyHint, destructiveHint, or idempotentHint. This forces LLMs to reason about side effects from descriptions alone, increasing error risk. Production pattern: all write tools should have destructiveHint=true; all idempotent tools (status checks) should declare idempotentHint=true.
Naming collision and redundancy: two tools generate microservices architecture guidance, microservices.arch.guidance.start (async with status polling) and generate_microservices_arch_guidance (blocking). Same pattern for data_pipeline. LLMs cannot distinguish when to pick which. Descriptions do not explain: blocking variant will hang the agent until completion; async variant returns immediately with a job_id for polling. No guidance on latency tradeoffs or when to prefer one over the other. This violates pattern:tool-distinctness, if two tools do the same thing differently, make the distinction unmissable in the name (e.g., start_microservices_guidance_async vs. generate_microservices_guidance_sync).
Error handling is absent. No tool provides recovery guidance, error classification, or actionable next steps. If a git clone fails (e.g., invalid URL, auth denied, network timeout), what should the LLM do? Retry? Ask the user to verify the URL? Call a different tool first? Without error responses that guide recovery, agents will either retry blindly or dead-end. Production pattern: every error must answer 'what should I do next?' Example: 'Invalid repo URL, verify format is https://github.com/user/repo or git@github.com:user/repo. If private, ensure credentials are configured.'
Parameter constraints are under-specified. Examples: 'depth' (optional integer) has no range hint, should it be 1-100? Unlimited? What happens if depth > repo history? 'branch' (optional string) has no validation, if branch doesn't exist, does the tool fail? Fall back to default? 'auth_mode' is an enum (https/ssh) but no description of when to use which or how credentials are injected. Production baseline: numeric params should state min/max; string params should state format, length, or pattern constraints in the description text (since LLMs cannot read JSON Schema pattern fields).
Job ID parameters are minimal and lack chaining context. Tools like git.repo.snapshot.status, cobol.repo.parse.status accept only a job_id string with no description of where to obtain it or what other context might be needed. If downstream tools need repo_url or workspace_id to process results, they are not returned in the status response. This breaks tool chaining, after polling for completion, the agent may not have the context needed for the next step without an extra lookup call. Production pattern: start/status pairs should return all IDs and references needed by downstream tools (pattern:tool-chain).