Loading ToolBench
Implementation of Model Context Protocol for scientific computing
Server has 6 tools with basic schemas and descriptions, but critical gaps undermine quality. Most tool descriptions are under 50 chars (well below the 194-char baseline for A+ tools). Parameter descriptions are minimal or absent. No output schemas documented. Error handling is generic (JSON-RPC error codes without recovery guidance). Tools like 'callTool' and 'submitJob' perform destructive operations (WRITE risk) but lack confirmation patterns or idempotency hints. Schema completeness varies: callTool and submitJob have typed parameters, but listResources and list lack input validation. No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Naming is inconsistent: 'listResources' vs 'list' both enumerate, creating ambiguity. Parameter naming lacks type suffixes (e.g., 'tool_name' vs 'tool_id'). No evidence of pagination support for list operations despite potential for large result sets.
Submit a job to a tool with script and core count
List HDF5 files matching a pattern in a specified path
List available resources
Get metadata for a specific HDF5 file
Read a specific dataset from an HDF5 file
Submit a job to SLURM with script, cores, memory, and job name
listResources has empty input schema ({}). Tool is non-functional without input validation.
Destructive tools (callTool, submitJob) lack confirmation/dry-run patterns and idempotency hints. No guidance on retry safety or side effects.
Tool descriptions are 20 - 45 chars, well below 194-char baseline. Descriptions lack WHEN/WHY context. E.g., 'List available resources' does not explain when to call it vs 'list' or what it returns.
No output schemas documented. LLMs cannot plan downstream calls or extract required fields. E.g., does 'read' return raw data or metadata? What fields are in the response?
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 36 | 2026-07-28+ | v2 |
Error responses use generic JSON-RPC codes (-32602, -32601, -32000) without recovery guidance. E.g., 'Missing action parameter' does not tell LLM what to do next.