Loading ToolBench
A Rust-based MCP server for Git and GitHub repository operations
The server defines 5 tools with basic descriptions and input schemas. However, critical gaps limit tool usability: (1) Output schemas are NOT documented anywhere, LLMs cannot predict return structure for planning; (2) Parameter descriptions are minimal and lack constraints/ranges/format details required by the rubric baseline (72 chars average); (3) Tool names are appropriately verb-prefixed (get_*) and clear, but descriptions lack WHEN/WHY context for LLM selection; (4) No error handling guidance, errors are raw strings without recovery hints; (5) Parameters lack enums, ranges, or validation rules despite accepting free-form strings. Each tool is READ_ONLY (good), but composition and context-chaining are weak. Per-tool scores average 42.
Fetches the changelog between two Git tags using GitHub's compare API
Fetches the content of a specific file from a GitHub repository
Fetches the file tree structure of a GitHub repository
Fetches the README file content from a GitHub repository
Retrieves Git tags from a repository with semantic version sorting
Output schemas not documented. Tool descriptions do not specify return structure, field types, or what data LLMs should expect. LLMs cannot plan multi-tool sequences or extract required IDs for chaining without documented return schemas.
Parameter descriptions lack format, range, and constraint details. 'branch' parameter described as 'optional string slice specifying the branch name (defaults to HEAD)' but does not clarify format (branch name regex?), length limits, or examples. Rubric baseline: avg param description 72 chars with explicit constraints.
Tool descriptions lack WHEN/WHY context for LLM selection. 'Retrieves Git tags from a repository with semantic version sorting' tells WHAT but not WHEN to use vs other tools, or how result connects to downstream actions (e.g., 'Use to find available releases before fetching changelog').
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 48 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 0 | - | v1 |
No error handling guidance in tool descriptions. When GitHub API fails or repository is invalid, tools return raw error strings ('Invalid GitHub URL', 'API Error: <status>') without recovery hints. Rubric pattern: 'Error responses must tell the LLM what to do next.'
No pagination or result-limiting documented for get_tags and get_changelog. If a repo has 1000+ tags or commits, responses could exhaust LLM context window. Rubric: 'cap results at reasonable limit (20-50) and offer pagination.' get_tags accepts 'limit' parameter but no guidance on default or max.
Tool composition weak: no tool chaining IDs documented. If get_tags returns tags but not a way to identify which tag to use for changelog, agent must infer or guess. Response structure for tags likely includes tag names only, unclear if changelog can accept any tag format returned.
Parameter 'link' expected to be GitHub URL but accepts any string. No enum, regex pattern, or format constraint enforced at schema level. LLMs may pass invalid URLs. Rubric: 'constrain inputs as enums or patterns.'
get_readme truncates content at 20,000 chars with '[TRUNCATED]' marker but no way to fetch full content. If user needs complete README, agent has no tool. Should either paginate or offer a separate tool to fetch by offset.