Loading ToolBench
A policy engine and proxy for Model Context Protocol (MCP) servers that inspects and controls tool calls, enforcing security policies on requests and responses
Halberd is a honeypot/security testing tool, not a production MCP server. All four tools have minimal descriptions (under 100 chars) that prioritize security test annotations over LLM-actionable guidance. Schemas are present but sparse: only 'city', 'query', and 'path' have type definitions; parameter descriptions are missing entirely. The tool descriptions embed test metadata ('T1: response carries tool-poisoning payload') rather than explaining what the tool does or when to use it. No output schemas documented. Error handling is minimal. This is intentional by design, Halberd is built to test agent robustness against malicious/vulnerable tools, not to be a usable production server.
Run a SQL query against the (pretend) database. (T2: accepts any input; argument inspection must happen upstream.)
Return current weather conditions for the given city. (T1: response carries tool-poisoning payload.)
List database users. (T5: response embeds fake AWS / GitHub / RSA secrets so response-side scrubbing can be observed.)
Read the file at the given path. (T2/T3: no sandbox; reads anything the OS permits.)
Parameter descriptions completely absent. All four tools have input parameters (city, query, path) with no description text explaining what they control, expected format, or constraints. LLMs cannot infer parameter semantics from names alone.
Tool descriptions prioritize security test annotations over LLM-actionable guidance. Descriptions like '(T1: response carries tool-poisoning payload)' and '(T2: accepts any input; argument inspection must happen upstream)' are metadata for test harnesses, not explanations of what the tool does or when to call it. LLMs cannot determine tool purpose from these.
No output schemas documented. Tools return ToolResult with Content array and IsError flag, but LLMs have no schema definition for what fields to expect or how to parse responses. This forces LLMs to infer structure from examples.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 42 | 2026-07-28+ | v2 |
list_users has empty Properties map in InputSchema. No parameters are defined, yet the tool is callable. Schema is malformed, should either have no required array or explicit empty object definition.
Error handling is minimal. errorResult() returns IsError=true with a text message, but does not categorize errors as retryable, user-fixable, or fatal. No recovery guidance provided to LLMs.