Loading ToolBench
MCP server that provides access to Siri shortcuts
The Siri Shortcuts MCP server has three tools with clearly defined names and basic schemas, but significant gaps in descriptions and parameter documentation. Tool naming is strong (verb-first, clear actions), but parameter descriptions are minimal, and output schemas are completely undocumented. Error handling exists but lacks recovery guidance. The server does not validate that shortcuts actually exist before attempting operations, and descriptions are too brief to guide LLM tool selection effectively.
List all available Siri shortcuts
Open a shortcut in the Shortcuts app
Run a shortcut by name or identifier (UUID) with optional input and output parameters
Parameter descriptions are too brief and lack actionable context. 'The name of the shortcut to open' and 'The name or identifier (UUID) of the shortcut to run' do not explain when to use names vs UUIDs, what happens if the shortcut does not exist, or how to discover available shortcuts. LLMs cannot disambiguate between parameters that sound similar without richer descriptions.
Output schemas are completely undocumented. The tools return ToolResult objects, but there is no schema documentation explaining what fields the LLM should expect in responses. For 'list_shortcuts', the response includes a 'shortcuts' array with 'name' and optional 'identifier' fields, this must be formally documented. For 'run_shortcut', the response structure (success, message, or error format) is unclear from the tool definition.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 16 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 39 | - | v1 |
Error responses lack recovery guidance. The code throws McpError with messages like 'Failed to list shortcuts: {error.message}' and 'Error listing shortcuts: {stderr}', but does not tell the LLM what to do next. For example, if a shortcut is not found, the error should suggest 'Call list_shortcuts() to see available shortcuts' rather than just reporting the failure.
Tool descriptions do not explain side effects or state mutations. 'open_shortcut' opens the Shortcuts app, a WRITE operation with visual side effects. 'run_shortcut' executes a shortcut, which may modify device state, create files, or trigger system actions. Descriptions must be explicit about what happens when the tool is called so LLMs understand idempotency and retry safety.
The 'run_shortcut' tool accepts both name and UUID but the parameter description does not explain how to use each or when to prefer one over the other. The optional 'input' parameter description says 'Can be text, or a filepath' but does not specify the format (plain text vs JSON), encoding (UTF-8), or maximum length. An LLM cannot reliably use this parameter without clearer constraints.
Tool composition is incomplete. If an LLM wants to run a shortcut but only knows its partial name, there is no guidance to call 'list_shortcuts' first to discover the exact name. The 'list_shortcuts' description does not state that it returns all available shortcuts, making it unclear what this tool is for until you inspect the implementation.
No input validation or sanitization visible in tool parameters. The code constructs shell commands using string interpolation (e.g., `exec(command, ...)` and `spawn(...)`), which is vulnerable to command injection if shortcut names contain shell metacharacters. Parameter descriptions do not specify allowed characters or constraints. The 'input' parameter to 'run_shortcut' is written to /tmp without validating content, risking injection attacks.
Descriptions are below the production baseline of 194 chars average. 'List all available Siri shortcuts' is 35 chars, well below the p10 threshold of 34 chars. 'Open a shortcut in the Shortcuts app' is 36 chars. These descriptions lack context about when to call the tool, what it returns, and when it is safe to retry.