Loading ToolBench
MCP server for automation using Apple Shortcuts
This server has two tools with visible schemas and descriptions, but significant quality gaps prevent a higher score. Both tools have actionable verb names (run_shortcut, list_shortcuts) following the verb_noun convention, which is good. However, descriptions are minimal (11-27 chars), parameter descriptions are sparse or missing, output schemas are completely undocumented, and there is no error handling guidance for LLMs. The run_shortcut tool accepts an optional 'input' parameter but does not document its format, type constraints, or when it should be used. The list_shortcuts tool has no parameters but its output structure (a newline-delimited string) is not formally documented. Neither tool documents what happens on failure or how an LLM should recover. Baseline comparison: average tool description should be 194 chars (p10=34, p90=392), these are at 27 and 28 chars respectively. Average param descriptions should be 72 chars, these tools provide 22 and 47 chars. The absence of output schema documentation violates a critical pattern: LLMs cannot plan downstream tool calls without knowing what fields to expect.
List all available shortcuts
Run a Shortcuts automation by name
Tool descriptions are critically short (11-28 chars). Minimum viable is 34 chars; baseline is 194 chars. LLMs cannot determine when or why to select a tool based on these descriptions.
Output schemas are completely undocumented. run_shortcut returns a text response, but LLMs don't know if it's freeform text, structured JSON, or what fields to extract. list_shortcuts returns a newline-delimited string, but the agent cannot parse it reliably without a formal schema.
The 'input' parameter in run_shortcut lacks a description explaining its format, expected content, length constraints, or character restrictions. Parameter descriptions should average 72 chars; this has none (empty string).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 49 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 35 | - | v1 |
No error handling guidance for LLMs. When a shortcut fails (e.g., not found, permission denied, timeout), the error message tells the LLM nothing actionable. The code returns a raw error message ('Failed to run shortcut: ...'), but does not categorize it as retryable, user-fixable, or fatal.
run_shortcut accepts a shortcut name as a free-form string, but does not validate it against the available shortcuts list. An LLM can pass a typo or non-existent name, and the tool will fail at execution time rather than catching the error early with guidance (e.g., 'Shortcut not found. Available shortcuts: ...') as per error classification pattern.
Security: The 'input' parameter is passed directly to a shell command (execSync) without sanitization. This is a command-injection risk if an LLM passes shell metacharacters or quotes. Input must be escaped using proper shell escaping (e.g., shlex.quote in Python, or equivalent in Node.js).
The 'name' parameter in run_shortcut also lacks proper escaping before being passed to execSync. A shortcut name containing spaces or quotes will break the command or cause injection.
No parameter constraints or validation. The 'name' parameter should document that it must be a valid shortcut name from the available list (a discovery dependency). The 'input' parameter should document its format (plaintext string, JSON, etc.) and any length limits.
Global state management: The availableShortcuts list is stored as a module-level variable and updated only when list_shortcuts is called. If shortcuts are added/removed between tool calls, run_shortcut will not see them. The tool should either call updateShortcutsList dynamically or accept that it may reference stale data.