Loading ToolBench
MCP server for PR template suggestions - Starter Code
Server has 3 tools with basic schemas and descriptions, but significant gaps prevent production use. Tool names lack clear action verbs (analyze_file_changes is acceptable, but get_pr_template and suggest_templates are generic). Descriptions exist but are brief (10-50 chars for some params). Critical issue: suggest_templates lacks parameter descriptions entirely, LLMs cannot infer what 'pr_description' and 'pr_type' mean. No output schemas documented. Error handling is minimal (returns JSON error strings, not actionable recovery guidance). Missing input validation, no enum constraints for pr_type despite listing valid values in code. The code references a Counter import that is never imported, indicating incomplete implementation.
Analyze file changes between branches using git diff, returning changed files, statistics, commits, and optionally the full diff content.
List PR templates dynamically from the templates directory, read their content, sort them alphabetically, and return templates with filename, type (derived), and file content.
Suggest appropriate PR templates based on PR description and type using keyword matching and token overlap scoring.
suggest_templates has NO parameter descriptions. 'pr_description' and 'pr_type' lack any guidance on format, constraints, or valid values. LLMs cannot infer meaning from names alone.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields (e.g., does get_pr_template return template_id, content, filename?). Breaks tool chaining.
pr_type parameter accepts free-form strings despite code defining valid enum values (bug, feature, docs, refactor, test, performance, security). No enum constraint declared in schema, invites hallucinated values.
Error handling returns raw JSON error strings without recovery guidance. 'Current directory is NOT a git repository' tells LLM nothing about what to do next (retry with working_directory param? call a different tool?).
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 43 | <=2025-11-25 | v2 |
Code references 'Counter' (from collections) in compute_token_overlap_score but never imports it. Implementation is incomplete and will fail at runtime.