Loading ToolBench
MCP server that monitors GitHub repos for engineering activity and converts code changes into sales leads
Server has 8 tools with basic schemas and descriptions, but significant gaps limit production readiness. All tools have descriptions (10-200 chars, baseline 194), and input schemas are present with type definitions. However, parameter descriptions are sparse or missing context. Output schemas are not documented. Error handling is minimal, tools return plain text without recovery guidance. No tool annotations (readOnlyHint/destructiveHint). Naming is verb-forward (watch_, list_, fetch_, analyze_, summarize_, classify_) but some names conflate multiple concerns (analyze_repo chains two operations; fetch_contributor_profiles mixes data retrieval with profile enrichment). No pagination support documented despite tools returning lists. Security: GitHub token and Anthropic API key are injected server-side (good), but no audit logging or permission gates visible.
Summarize recent GitHub activity for a repository and classify it into SaaS domain categories in a single call. Chains summarize_activity → classify_signal and returns both the synopsis and the ranked domain list.
Map an activity synopsis to SaaS domain categories with confidence levels. Returns a ranked list of {domain, confidence, reasoning} objects.
Search for recent news about the company behind a GitHub org: funding, product launches, technical blog posts, hiring surges, and partnerships. Optionally provide org_domain (e.g. "stripe.com") to improve search targeting.
Fetch GitHub profile data for the top contributors of a repository: name, company, location, bio, follower count, and public org memberships.
Fetch recent commits, pull requests, and issues for a repository. Uses last_activity_hash to skip repos with no new commits. Updates last_checked and last_activity_hash in the registry.
Output schemas not documented. Tools return TextContent or JSON but LLMs cannot infer field structure for downstream chaining. E.g., fetch_repo_activity returns activity JSON but schema is not declared.
analyze_repo conflates two responsibilities: summarize_activity + classify_signal. Violates single-responsibility principle. LLM cannot compose these independently.
No pagination support. list_watched_repos and tools returning lists lack limit/offset/cursor parameters. Large result sets will blow context window.
Error handling returns plain text without recovery guidance. E.g., 'No new activity' or 'not found' errors do not tell LLM what to do next or whether to retry.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 52 | 2026-07-28+ | v2 |
List all repositories currently in the watched-repo registry.
Fetch recent GitHub activity for a repository and use Claude to produce a 2-3 sentence technical synopsis of what the team is actively building.
Add a GitHub repository to the watched-repo registry so it can be monitored for engineering activity.
No tool annotations. Tools lack readOnlyHint/destructiveHint/idempotentHint. watch_repo is destructive (modifies registry) but not marked. LLM cannot reason about side effects.