Loading ToolBench
Security primitives for MCP servers — isError-compliance, JSONL audit, JSON-schema input validation, output sanitization, OAuth 2.1 scope enforcement
mcp-shield demonstrates solid tool design with complete schemas, clear descriptions, and good naming conventions. All 6 tools follow verb_noun patterns (audit_tail, audit_summary, red_team, scan_input, patterns_list, sanitize_preview). Schemas are well-formed with proper type constraints, minLength/maxLength, and additionalProperties=false. Descriptions are substantive (100-250 chars) and include use-case guidance. However, parameter descriptions are sparse or missing in some tools (e.g., shield_red_team's 'extra_patterns' lacks detail on regex format expectations). Output schemas are not documented, LLMs cannot predict response structure. Error handling is minimal; tools return JSON but lack recovery guidance. No tool annotations (readOnlyHint) despite all being read-only operations.
Aggregate counts over a JSONL audit log: total calls, error rate, top tools. Use when: "how healthy is the MCP fleet?", "what's the error rate?". Example: path="/var/log/mcp-shield/audit.jsonl".
Return the last ``n`` records from a JSONL audit log. Use when: "what did the agent just do?", "did anything fail?". Example: path="/var/log/mcp-shield/audit.jsonl", n=50.
List the prompt-injection patterns the sanitizer detects. Use when: "what does this tool actually look for?".
Run a curated set of injection payloads through the sanitizer. Use when: "is the sanitizer actually catching things?", "compliance evidence for security review". Example: strict=False (default; reports hits without raising). Each result row shows ``{"input", "caught", "pattern"}``. ``caught`` is True iff the sanitizer matched any pattern.
Preview the sanitizer's output on a text sample.
Output schemas not documented. LLMs cannot predict response structure (e.g., audit_summary returns counts/error_types/durations but this is not declared in tool definition).
Parameter descriptions incomplete. shield_red_team's 'extra_patterns' lacks format guidance (regex syntax, escaping rules). shield_patterns_list has empty properties object with no description of what it returns.
No tool annotations. All 6 tools are read-only (Risk: READ_ONLY marked) but lack readOnlyHint annotation in schema. Agents cannot infer safety/idempotency from tool definition alone.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | 2026-07-28+ | v2 |
Validate an ``args`` dict against a JSON schema; return OK or first error. Use when: "is my agent's call valid before I send it?". Example: args={"q":"hi"}, schema={"type":"object","properties":{"q":{"type":"string"}}}. Returns ``{"valid": true}`` or ``{"valid": false, "error": "..."}`. Never raises — the tool surfaces validation results as data.
Error handling lacks recovery guidance. audit_summary returns {'error': 'log not found'} but does not suggest next steps (e.g., 'Check MCP_SHIELD_AUDIT_LOG env var' or 'Call shield_patterns_list to verify server is running').
shield_patterns_list description is minimal ('List the prompt-injection patterns the sanitizer detects.') and lacks context on when/why to call it or what structure is returned.