Loading ToolBench
Three tools with clear action verbs and visible Zod schemas, but significant gaps in parameter descriptions and output documentation. Tool names follow verb_noun convention (get_*), which is good. All three tools have non-empty descriptions (174-189 chars, within baseline range). However, parameter descriptions are minimal or missing clarity. Input schemas are present via Zod but lack contextual documentation. Output is unstructured text rather than typed JSON objects, and there is no documented output schema for LLMs to plan downstream calls. Error handling exists but provides generic messages without actionable recovery steps. No parameters expose secrets (GITHUB_TOKEN is env-injected), which is correct. Overall, the server implements basic structure but lacks the polish and completeness required for A/B-level scores.
Get content of a specific file from a GitHub repository
Get all files from a GitHub repository to use as context
Get the structure of a GitHub repository
Output schema undocumented. Tools return wrapped text content via MCP ContentBlock format, but no structured JSON schema is documented for LLMs to understand or plan downstream calls. get-repo-context returns formatted file listings; get-file-content returns raw content; get-repo-structure returns a file listing. LLMs cannot extract structured data (e.g., file count, repository metadata) for use in subsequent agent reasoning.
Parameter descriptions lack actionable constraints. 'maxFiles' defaults to 50 but description does not specify min/max bounds or why 50 is the default. 'fileExtensions' example in description ('js', 'ts', 'md') may be literal-copied by LLMs per the rubric. 'excludePaths' defaults to ['node_modules', 'dist', 'build'] but description does not explain the rationale or how to override safely.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 49 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 45 | - | v1 |
Error handling is generic and non-actionable. All three tools catch errors and return 'Error fetching [resource]: [error message]'. No guidance on what the LLM should do next (retry? Try a different repo? Check GITHUB_TOKEN?). The warning on startup about GITHUB_TOKEN being missing is logged to stderr, not surfaced to the agent via tool responses, so agents cannot adapt.
No pagination or result limiting for get-repo-context. The tool fetches all files in a repository (potentially thousands) and filters to maxFiles = 50 by default. For very large repos, getAllFiles() makes recursive API calls and could exhaust rate limits or context window. No next_cursor or page parameter to allow agents to explore large repositories incrementally.
Tool composition issue: get-repo-context and get-file-content may be better combined into a single tool or separated more cleanly. An agent that wants to 'fetch the README' must call get-repo-context (slow, fetches 50 files) or guess the path and call get-file-content. get-repo-structure returns only paths, not metadata. No tool returns a combined summary (file count, language breakdown, total size).