Loading ToolBench
A Model Context Protocol server that integrates Llama language model with GitHub repository file listing capabilities. Provides context retrieval and GitHub API integration through FastAPI endpoints.
Server has 3 tools with basic FastAPI definitions. All tools have descriptions (10-100 chars), but descriptions are generic and lack LLM-optimization guidance. Input schemas are present via Pydantic models but lack parameter-level descriptions and constraints. No output schema documentation. Error handling is minimal, no recovery guidance or error classification. Tool naming is reasonable (get_context, list_files, health_check) but descriptions don't explain WHEN to use each tool or dependencies between them. No tool annotations (readOnlyHint, etc.). Composition is weak: get_context conflates GitHub listing with Llama querying in one tool, violating single-responsibility principle.
Fetch context information from the Model Context Protocol server. Handles both GitHub repository file listing requests and general Llama model queries based on the query text.
Health check endpoint that verifies connectivity to both Llama API and GitHub API. Returns status of both services.
Dedicated endpoint to list files from a GitHub repository. Parses GitHub repository URL and returns formatted list of files and directories.
get_context conflates two distinct operations (GitHub file listing + Llama querying) in one tool. Violates single-responsibility principle. LLM must infer intent from query_text string parsing, not tool name.
Parameter descriptions missing or trivial. 'query_text' lacks guidance on format/examples. 'user_id' and 'session_id' marked optional but no explanation of when they're needed. 'additional_context' is a free-form object with no schema or usage hints.
No output schema documentation. ContextResponse.context_elements is a list of dicts with unspecified structure. LLM cannot predict what fields to extract or how to chain results to downstream tools.
Error handling provides no recovery guidance. GitHub API errors return raw error messages without suggesting alternatives (e.g., 'URL invalid, try search_repos first'). No error classification (retryable vs fatal).
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 48 | <=2025-11-25 | v2 |
Tool descriptions are under 100 chars and lack LLM-optimization. 'Fetch context information' is vague. No explanation of when to call get_context vs list_files, or dependencies on health_check.