Loading ToolBench
Java-based MCP server that loads and executes tools from JSON configuration files or HTTP endpoints
This Java MCP server exposes 8 tools with basic HTTP/command execution. The source reveals a generic tool executor that loads tool configurations from JSON files. However, critical issues undermine definition quality: (1) Tool definitions themselves are not visible in the provided source code, only the Java execution framework is shown. The actual tool schemas, descriptions, and names are declared to exist in external JSON files (tools/*.json), which means tool definitions are INFERRED rather than directly visible. Per HARD SCORING RULE, this caps per-tool scores at 50. (2) The descriptions provided in the problem statement are present but brief (34-104 chars), ranging from basic to reasonable but lacking LLM-optimized detail. (3) Input schemas are visible in JSON form in the problem statement and appear well-typed with required parameters, but without seeing the actual source files, schema completeness cannot be fully verified. (4) No parameter descriptions are visible in the schema definitions provided, only parameter names and types. (5) No error handling guidance, recovery hints, or validation messages are evident in the executor code. (6) No tool annotations (readOnlyHint, destructiveHint) are present. Overall: functional but minimal definitions with significant gaps in description depth, parameter documentation, and error guidance.
Evaluates mathematical expressions safely using AST parsing, supporting basic arithmetic operations
Compares values using max, min, or sort operations on arrays
Fetches and strips HTML content from a URL, returning plain text with optional length limit
Returns current time in ISO format and Unix timestamp, optionally for a specified timezone
Fetches current weather information for a city using OpenWeatherMap API
Lists files and directories in a specified path within the 'data' root directory
Reads file content from the 'data' directory with optional byte limit
Performs web search using a SearXNG instance, returning title, URL, content, and engine for each result
Tool definitions not visible in source code, only inferred from JSON file loading. Actual tool schemas, descriptions, and configurations exist in external tools/*.json files not provided. Per HARD SCORING RULE, tools defined outside visible source are capped at 50 per-tool score.
Parameter descriptions are completely missing from all tools. Schemas show parameter names and types but no description fields. LLMs cannot infer what 'expression' means in calculator without documentation. Per HARD SCORING RULE, lack of parameter descriptions cannot exceed 30 for schema score.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 47 | <=2025-11-25 | v2 |
No error handling guidance or recovery hints in ToolExecutor. Errors are caught but minimal information is returned to the LLM. A failed HTTP call returns a generic exception; no indication whether to retry, check parameters, or escalate.
No tool annotations present. Tools are marked READ_ONLY but this is metadata in the problem statement, not in the actual tool definitions or ToolConfig class. The server does not declare readOnlyHint, destructiveHint, or idempotentHint in tool definitions.
Tool descriptions are brief (34 - 104 chars) and lack LLM-optimized detail. Example: 'Evaluates mathematical expressions safely' is functional but does not explain WHEN to use it vs alternatives, what formats are supported, or what the return value contains. Rubric baseline: A+ tools average 194 chars with strong context.
No output schema documentation. Tools return Map<String, Object> in ToolExecutor.call() wrapped as {content: [{type: text, text: ...}]}, but the actual structure of the result object is not documented. LLMs cannot plan downstream calls without knowing what fields to expect.
No validation of input parameters. ToolExecutor accepts any Map<String, Object> as arguments and passes it directly to backends. No enforcement of required fields, type validation, or constraint checking (e.g., maxLength must be positive). Invalid inputs produce silent failures or generic errors.
getWeather includes no API key parameter but requires OpenWeatherMap API access. If credentials are hardcoded in the backend Python script or injected via environment, this is correct (pattern:secret-injection), but it is not documented. Unclear whether credentials are secure or embedded in tool definitions.
readFile and listFiles expose a 'data' directory without explicit access control or path traversal protection in the visible code. The descriptions mention 'within the data directory' but no validation is shown to prevent '../' escapes. Path traversal via crafted 'path' parameters could read arbitrary files.
webSearch depends on an external SearXNG instance with a default URL (http://127.0.0.1:8888). If the instance is not running or the URL is wrong, no helpful error message guides the LLM. Additionally, allowing users to override searxng_url via parameter could enable SSRF attacks if not validated.