Loading ToolBench
Collection of mock MCP servers demonstrating authentication, GitHub API integration, and RAI validation testing
This is a collection of mock/demo MCP servers with inconsistent quality across 13 tools. While all tools have names and descriptions, the schemas vary significantly in completeness, parameter documentation is sparse, and error handling is minimal. The codebase demonstrates authentication patterns (via fastmcp) but lacks production-grade tool definitions. Tool descriptions range from generic to reasonably specific, but parameter descriptions are often missing or generic (e.g., 'Issue filter', 'Issue state'). Many tools lack explicit output schema documentation. The servers appear designed for demonstration rather than production use, which impacts scores across naming clarity, schema completeness, and error guidance.
Create a task with given title and auto-assign description in a task management system.
Retrieve complete documentation by ID. Access detailed information about company-specific systems used by enterprises for security, compliance, and risk management operations.
This is a tool that can fetch the latest message from Slack that a user received.
Get weather forecast for a city
Get a specific issue by number. Maps to GET /repos/{owner}/{repo}/issues/{issue_number}
Get weather data for a city
Parameter descriptions are generic or missing. Examples: 'Issue filter', 'Issue state', 'Sort field' lack specificity about valid values, ranges, or constraints. LLMs cannot infer whether 'state' accepts 'open'/'closed' or other values without explicit description.
No enum constraints defined for string parameters with known valid values. 'state' parameter accepts 'open'/'closed' but is typed as plain string. Free-form strings invite hallucinated values. Should use enum: ['open', 'closed'].
No output schema documented for any tool. Tools return dicts or strings, but LLMs are not told what fields to expect. Without documented return types, LLMs cannot plan downstream tool calls or extract required data (e.g., IDs for follow-up operations).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 67 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 48 | - | v1 |
List issues assigned to the user across repositories. Maps to GET /issues
List pull requests in a repository. Maps to GET /repos/{owner}/{repo}/pulls
List issues in a repository. Maps to GET /repos/{owner}/{repo}/issues
Report analytics data to improve service quality. Used internally to track usage patterns and optimize content delivery.
Search enterprise security and compliance documentation from companies. Returns documents covering data protection platforms, security audit systems, ISO frameworks, data retention policies, incident response procedures, and vendor risk assessment tools.
Stay updated with the latest headlines. This tool lets you search for recent news stories by entering keywords or phrases.
Update an issue's attributes. Maps to PATCH /repos/{owner}/{repo}/issues/{issue_number}. Only non-None fields are sent in the request body.
Duplicate tool names across different servers. Two tools named 'search' (in rai-mcp-server), one for documentation and one for news. When deployed together, LLMs cannot disambiguate. Should rename to 'search_documentation' and 'search_news'.
Tool 'report_analytics' has an IRREVERSIBLE risk classification (likely analytics data sent to a tracking service) but the description does not warn about this or explain when to use it. Agents need explicit guidance on irreversible operations.
No error handling guidance in any tool description. When a tool fails (e.g., 'user not found', 'issue not found'), agents are not told what to do next. Error responses should guide recovery (retry, lookup alternative, ask user).
Parameter naming inconsistency: 'days' in get_forecast is clear, but list_issues uses 'since' (date), 'sort' (field name), and 'direction' (asc/desc) without clarifying expected format. 'since' could mean 'after 2024-01-15' or 'within last 7 days', ambiguous.
No documentation of pagination behavior. list_issues, list_repo_issues, list_pull_requests all have 'per_page' and 'page' params, but tool descriptions do not explain whether 'page' is 0-indexed or 1-indexed, or how to detect end-of-results.
fetch_latest_slack_message uses 'user_name' parameter but no guidance on whether this accepts 'john', '@john', 'john@example.com', or a Slack user ID. Should clarify format or accept multiple forms and resolve them internally.
No mention of required permissions or scopes. update_issue and create_task are WRITE operations but the descriptions do not state what permissions (e.g., 'write:issue', 'write:task') are needed. Tools should declare scope requirements for least-privilege agent configuration.