Loading ToolBench
This collection has significant definition quality gaps. The demo server (3 tools) lacks proper MCP SDK integration and uses a custom stdin parser instead of the official SDK. The weather server (2 tools) properly uses the MCP SDK with Zod schemas, but descriptions are minimal (10-30 chars, well below the 50-200 char baseline for A+ tools). All 5 tools have descriptions, but they are terse and lack context about when to use them or what they return. Parameter descriptions exist but are sparse. No output schemas are documented. No error handling guidance is provided. The demo tools have no visible schema definitions in the source code, only inferred from the controller logic.
Get current weather for a location
Get custom greeting with name parameter
Get default greeting
Get 5-day weather forecast for a location
Get greeting using custom template
Demo server does not use MCP SDK, implements custom stdin JSON parser instead of McpServer. Tool definitions are inferred from controller logic, not explicitly registered with the SDK.
All tool descriptions are under 50 characters and lack context. 'Get default greeting' does not explain when to use it, what it returns, or how it differs from get-custom-greeting. Baseline for A+ tools is 50-200 chars.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract fields without knowing the response structure. Weather tools return {content: [{type, text}]} but this is not documented in tool definitions.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 35 | 2026-07-28+ | v2 |
Demo tools lack error handling guidance. No recovery hints, no categorization of errors as retryable vs fatal. Controllers throw generic 'Internal server error' without actionable context.
Weather API key exposed in source code (src/weather/config/index.ts). Credentials must never appear in tool parameters or config files, use environment variables or vault injection.