Loading ToolBench
Multi-protocol MCP server suite providing tools for database access (PostgreSQL, MySQL, MSSQL), filesystem operations, GitHub integration, Jira integration, and browser automation via Playwright.
ServeRay defines 20 tools across 5 domains (DB, filesystem, GitHub, Jira, web). All tools have descriptions (10 - 200 chars, baseline 194) and explicit input schemas with typed parameters. Naming follows verb_noun convention (query, list_*, describe_*, get_*, search_*, fetch_*, extract_*, capture_*). However, several tools lack parameter descriptions (e.g., 'limit' in list_databases has a description, but many optional params are minimally documented). Output schemas are not explicitly documented in the source, responses are inferred from code (e.g., QueryResult.JSON, []map[string]any). Error handling returns text messages but lacks recovery guidance (e.g., 'User not found. Try search_users()' pattern absent). Security is strong: no credentials in params, allowlist-based access control for repos/roots/projects. Composition is clean, each tool has one responsibility, and tool chains are supported (e.g., list_tables → describe_table). Overall: solid definitions with good naming and descriptions, but missing output schema documentation and recovery-oriented error messages.
Capture a screenshot of an allowlisted URL.
Describe the columns for a table. Use schema.table when needed.
Execute a single write statement. This tool only appears when explicit unsafe write mode is enabled.
Extract links from an allowlisted URL.
Navigate to an allowlisted URL and return bounded page text.
Get details for a specific Jira issue by key.
Get details for a specific GitHub issue.
Output schemas not documented. Tools return QueryResult.JSON, []map[string]any, or text, but LLMs cannot see the structure of returned fields. This forces agents to guess what data is available and risks failed downstream tool calls.
Error responses lack recovery guidance. When a query fails or a resource is not found, the tool returns a plain error message (e.g., 'User not found') without suggesting next steps (e.g., 'Try search_users() with a partial name'). This forces agents to reason about recovery without hints.
Duplicate tool name 'get_issue' (GitHub and Jira). LLMs cannot distinguish between them without additional context. Rename one to 'get_github_issue' or 'get_jira_issue' to avoid ambiguity.
Inferred effective spec: 2025-06-18+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | 2025-06-18+ | v2 |
List the Jira project keys this server is allowed to access.
List the GitHub repositories this server is allowed to access.
List configured filesystem roots and their aliases.
List visible databases and basic metadata.
List directory entries under a configured root.
List issues in an allowlisted GitHub repository.
List pull requests in an allowlisted GitHub repository.
List visible schemas.
List visible tables and basic table metadata.
Execute a read-only SQL query. Safe mode blocks writes, multi-statement input, and dangerous keywords.
Read a bounded slice of a text file under a configured root.
Search issues within an allowlisted Jira project using structured filters.
Stat a file or directory under a configured root.
Optional parameters lack constraints. 'limit' appears in 8+ tools but has no min/max bounds documented. Unbounded numeric parameters let LLMs pass absurd values (e.g., limit=999999) that break APIs or cause timeouts.
No pagination support documented. Tools like list_issues and search_issues accept a 'limit' but do not expose offset/cursor or total_count. Large result sets risk context window exhaustion.