Loading ToolBench
MCP Server using the official Java SDK
Single tool with minimal definition quality. The tool 'get_logs' has a verb-based name ('get_') and an input schema present, but the description is extremely terse (24 chars), barely meeting the minimum threshold. The schema is present but lacks parameter descriptions entirely, the 'source' enum parameter has no description explaining what each value does, when to use datadog vs elastic vs file, or what the expected output looks like. No output schema is documented. The tool implements mock/file-based responses but does not explain error handling, pagination, or field structure of the returned logs. No per-parameter guidance on format, constraints, or dependencies. Error messages in code are generic ('Unknown source'). The implementation reads files without documented limits or structure. Across the 549-baseline production tools, this tool would rank in the bottom quartile for description completeness and parameter clarity.
Return logs from source (mock/file)
Tool description is too terse (24 chars: 'Return logs from source (mock/file)'). Does not explain WHAT the tool returns, WHEN to use each source (datadog vs elastic vs file), or any prerequisites. Baseline is 194 chars; this is 87% below target.
The 'source' parameter enum ['datadog','elastic','file'] has NO description. LLM cannot reason about which source to choose or what each returns. Parameter descriptions are mandatory, 100% of A+ tools have them.
No output schema documented. Tool returns McpSchema.TextContent but LLM has no way to know the structure of the returned logs (format, fields, line-delimited vs JSON, timestamp format, etc.). 100% of A+ tools document return types.
No pagination support or result limits documented. Code snippet shows 'skip(Math.max(0, Files.lines(p).count() - 1000))', capping at 1000 lines, but this limit is not exposed in the tool description or schema. If logs exceed 1000 lines, LLM has no way to request earlier/later ranges. No limit parameter in schema.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 42 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 25 | - | v1 |
Error handling is minimal. Code returns bare error strings ('Unknown source: X', 'File not found: X', 'Error reading file: X') without recovery guidance. Per pattern:recovery-guide, errors must tell LLM what to do next. E.g. 'Unknown source, must be one of: datadog, elastic, file. Call with a valid source.' instead of 'Unknown source: X'.
Dependencies on environment variables (LOGS_FILE_PATH) are not documented in the tool description. LLM cannot know that file source requires this env var to be set. Should state 'Requires LOGS_FILE_PATH environment variable' in description or parameter hints.
No indication of what data each source (datadog, elastic, file) actually returns. Mock responses are hard-coded in code ('sampleDatadog()', 'sampleElastic()') but tool description does not explain whether these are real integrations or stubs, or what the user should expect.