Loading ToolBench
MCP server for tidymodels GitHub information and code generation
The tidymodels-server has 7 tools with visible definitions in js/index.js. All tools have descriptions and parameter schemas are present in the code. However, the implementation shows critical gaps in schema completeness, parameter descriptions, output documentation, and error handling guidance. Most tools accept optional parameters with minimal validation hints. The 'generate_r_code' tool is the only one with a constrained enum (template_type), but most other tools lack parameter constraints. Descriptions are adequate (60-120 chars) but lack actionable guidance for LLM selection. No output schemas are documented, forcing LLMs to infer response structure. Error handling exists (McpError thrown) but provides minimal recovery guidance. The server performs read-only operations on GitHub APIs, which is low-risk, but composition and chaining support is weak, tools return raw API responses without ensuring downstream tools can consume them. Per-tool scores average 38/100.
Generate R code templates for tidymodels workflows based on template type and request description
Get the raw content of a file from a tidymodels repository
Get the contents of a directory or file in a tidymodels repository
Get reference information for tidymodels R packages including DESCRIPTION files and README content
List all repositories in the tidymodels GitHub organization
Search for code across the tidymodels organization
Search for function documentation across tidymodels repositories using roxygen blocks
No output/return schemas documented. Tools return raw GitHub API responses or parsed R package metadata, but LLMs cannot infer response structure. For example, list_tidymodels_repositories likely returns an array of repo objects, but the exact fields (name, description, stars, url, language, created_at, etc.) are not declared. This forces LLMs to reason blindly about downstream field availability and risks broken tool chains.
Parameter descriptions lack actionable constraints and validation rules. For example, 'query' in search_code has description 'The search query' but provides no guidance on minimum/maximum length, allowed characters, or examples. 'path' in get_repository_content is 'The path within the repository (default: root)' but doesn't specify if it should start with '/', how deep it can go, or if it supports glob patterns. These missing constraints force LLMs to guess valid input formats.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 42 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 0 | - | v1 |
Error responses provide minimal recovery guidance. Code throws McpError(ErrorCode.InternalError, ...) with messages like 'Failed to fetch repositories: {error}' or 'Failed to search code: {error}'. These do not tell the LLM what to do next: Is this retryable? Should the user check the GITHUB_TOKEN environment variable? Is the query malformed? Bare error messages leave agents unable to self-correct.
Tool composition is weak. Tools like get_repository_content and get_file_content both exist, but their distinctions are unclear from descriptions alone ('Get the contents of a directory or file' vs 'Get the raw content of a file'). No tool documents what IDs or references it returns that downstream tools can use. For example, if search_code returns file paths, does generate_r_code accept a file path parameter, or must the user call get_file_content first? The tool chain is implicit and error-prone.
Missing pagination and result limits. list_tidymodels_repositories fetches up to 100 repos via GitHub API, but the tool description does not mention this limit. search_code is similarly capped at 100 items per request, but the tool description gives no hint. Without explicit limits and pagination guidance (offset, limit, total_count, next_cursor), LLMs cannot handle large result sets and may exhaust context windows or miss data.
Tool descriptions do not clarify when to use each tool or include dependency hints. For example, get_tidymodels_r_reference's description is 'Get reference information for tidymodels R packages...' but does not say when to call it vs list_tidymodels_repositories. Are they complementary? Does the user need to list repos first, then fetch detailed reference info? The description should include: 'Call list_tidymodels_repositories first to see all packages, then use this tool to get detailed R package metadata (DESCRIPTION, README).'
get_tidymodels_r_reference and search_function_documentation are poorly differentiated. Both provide information about R packages. The distinction is unclear: does search_function_documentation search within DESCRIPTION/README (like get_tidymodels_r_reference), or does it search roxygen blocks in source code? The description mentions 'roxygen blocks' but the code snippet is incomplete. This naming ambiguity will cause LLMs to conflate and misuse the tools.
generate_r_code is underspecified. The 'request' parameter accepts free-form text ('Description of the code to generate'), but there is no guidance on how detailed the request must be, what language features are supported, or how the tool handles ambiguous requests. The description states 'Generate R code templates' but does not explain what templates are available beyond the enum (recipe, model, workflow, tuning, evaluation). Without examples or constraints, LLMs will make poor requests.