Loading ToolBench
This is a Python evaluation framework, not an MCP server. It exposes 11 tools via a CLI interface with custom executors and scorers. Tool definitions are visible in core/executors.py and core/scorers.py. Most tools have descriptions (8-12 sentences), but parameter schemas are incomplete: many lack type constraints, enums, or validation rules. Descriptions are verbose (100-200+ chars) but lack actionable constraints. No error handling guidance, no output schemas documented, no security considerations. The framework is designed for local testing, not agent integration. Naming is reasonable (verb_noun style: PythonFunc, HTTPEndpoint, ExactMatch), but parameter documentation is sparse. This scores as a D-range tool suite: functional but not production-ready for agent use.
Run a sequence of executors in order. Each executor receives the output of the previous one as its input. The final output is returned. Use this to evaluate multi-step agent pipelines end-to-end.
Run a subprocess. input is interpolated into the command via {input}.
Passes if all required strings appear in actual output.
Wrap any function: fn(actual, expected) -> bool or ScorerResult.
Passes only if actual == expected (stripped).
POST input as JSON to a REST endpoint, return response body as str. If input_key is set, wraps input as {input_key: input} before posting.
Asks an Ollama model to judge whether the actual output satisfies the rubric. rubric: a plain-English description of what a correct answer looks like. If omitted, uses expected output as the reference. Requires Ollama running at localhost:11434.
No input schema validation or type constraints. Parameters like 'timeout' (integer) lack min/max bounds; 'model' (string) lacks enum of valid Ollama models; 'method' (string) lacks HTTP verb enum. LLMs will pass invalid values.
No output schemas documented. Tools return strings or objects, but LLMs cannot plan downstream calls without knowing response structure. E.g., HTTPEndpoint returns 'response body as str', what fields? What format?
No error handling guidance. Tools fail silently or return raw exceptions. E.g., 'Subprocess timeout in seconds', what happens if timeout is exceeded? Does it retry? Fail? Return partial output?
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 37 | 2026-07-28+ | v2 |
Call a tool on a running MCP server via its HTTP bridge (mcp-bridge). Assumes mcp-bridge is running and exposes tools at: POST http://localhost:{port}/tools/{tool_name}
Send a prompt to an Ollama model and return the response text. template: use {input} as placeholder for the test case input. system: optional system prompt. Requires Ollama running at localhost:11434.
Call any Python function. Output is str(result).
Embeds actual and expected via Ollama's nomic-embed-text model, computes cosine similarity, passes if above threshold. Requires Ollama running at localhost:11434.
Parameter descriptions lack actionable constraints. 'Ollama model name (default llama3.2)' does not specify valid model names, format, or how to discover available models. LLMs will hallucinate model names.
No security considerations. HTTPEndpoint accepts arbitrary URLs and headers; MCPTool accepts Bearer tokens as parameters (should use server-side injection). CLI runs subprocesses without sanitization against command injection.