Loading ToolBench
GitHub Chat MCP server for analyzing GitHub repositories
Two tools with decent verb-based naming (index_, query_) and basic descriptions present. However, both tools lack critical description depth, parameter validation guidance, and output schema documentation. The conversation_history parameter has no type constraints (array of what?). Return types are undocumented, LLMs cannot plan downstream calls or extract needed fields. Error handling is generic ('Error: ...') without recovery guidance. No input validation rules documented (e.g., URL format, question length). Dependencies between tools (must index before query) are mentioned in descriptions but not formalized. Overall, definitions fall in the 'fair to poor' range, present but incomplete for production LLM agents.
Index a GitHub repository to analyze its codebase. This must be done before asking questions about the repository.
Ask questions about a GitHub repository and receive detailed AI responses. The repository must be indexed first.
Output schemas completely undocumented. Neither tool declares what fields the response contains. LLMs cannot plan chained calls or extract structured data. For index_repository: does it return a status code, repository metadata, or just a message? For query_repository: what fields does the formatted_response contain? Are sources guaranteed? This forces LLMs to make assumptions and wastes tokens parsing free-text output.
conversation_history parameter lacks type constraints. Declared as 'Optional[List[Dict[str, str]]]' but the structure is never documented. What keys are expected in each dict? Must it be ['role', 'content']? Can it contain other fields? Undocumented nested structures invite LLM errors and misuse.
Error handling is generic and non-actionable. Both tools return plain strings like 'Error: ...' or 'Error querying repository: {response.text}'. No recovery guidance. If a repository fails to index, should the LLM retry? Try a different URL format? Give up? Raw API error text is not actionable for LLM agents.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 46 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 0 | - | v1 |
No input validation rules documented. The descriptions mention URL format ('https://github.com/username/repo') but do not state min/max string lengths, character restrictions, or what happens if the repo does not exist. Validation errors are caught but returned as generic 'Error: ValueError(...)' instead of actionable messages like 'Invalid repo URL format; expected https://github.com/username/repo'.
Tool descriptions lack WHEN and WHY guidance. The index_repository description 'Index a GitHub repository to analyze its codebase. This must be done before asking questions.' states the prerequisite but does not explain when indexing is complete (how long does it take?), what triggers re-indexing, or what the agent should do if indexing fails. Similar gaps in query_repository.
Missing pagination and result limits. If query_repository returns many 'sources' or a long answer, there is no limit or pagination parameter. Large responses waste tokens and degrade LLM reasoning. No documentation of max answer length or how many sources are returned.