Loading ToolBench
Server has 18 tools with consistent verb_noun naming (launchd_list, launchd_start, etc.). All tools have descriptions (avg ~80 chars, within baseline 34-392). Input schemas are present with type definitions and parameter descriptions. However, output schemas are not documented, responses are inferred from code but not formally specified in tool definitions. Error handling exists but lacks recovery guidance. No tool annotations (readOnlyHint/destructiveHint) despite clear risk stratification in metadata. Missing idempotency guarantees for write operations.
Create a new launchd service with the specified configuration
Delete a launchd service by removing its plist file
Disable a launchd service by setting the Disabled key in its plist
List all available launchd domains
Enable a launchd service by removing the Disabled key from its plist
Force reload a launchd service by unloading and reloading its plist
List launchd services with optional filtering by domain, status, and label pattern
Output schemas not documented. Tool definitions lack return type specifications. LLMs cannot plan downstream calls or extract required fields (e.g., does launchd_list return label, pid, status fields?). Responses are inferred from code but not formally declared.
Write/destructive tools lack recovery guidance in error responses. When launchd_delete or launchd_plist_write fails, error messages do not guide the LLM on retry strategy, compensation, or next steps. No confirmation/dry-run pattern for irreversible operations.
Tool annotations missing. Despite clear risk stratification (READ_ONLY, WRITE, DESTRUCTIVE), tools lack readOnlyHint, destructiveHint, and idempotentHint annotations in schema. LLMs cannot infer safety properties without explicit hints.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | 2026-07-28+ | v2 |
Load a launchd service plist and start the service
Retrieve recent log entries for a launchd service
Open the System Settings GUI for a launchd service
Read and parse the plist file for a launchd service
Write content to a launchd service plist file
Restart a launchd service
Execute an arbitrary shell command with output capture
Start a launchd service
Get detailed status information for a specific launchd service
Stop a launchd service
Unload a launchd service plist and stop the service
launchd_shell_command is a generic escape hatch. Accepts arbitrary shell commands with no validation, sanitization, or scope limits. Violates principle of least privilege and enables command injection if LLM is prompt-injected. Should be removed or heavily gated.
Descriptions for write/control tools are too brief (<60 chars). launchd_start, launchd_stop, launchd_restart, launchd_load, launchd_unload, launchd_delete lack context on side effects, prerequisites, or when to use each. Baseline for A+ tools is 50-200 chars with clear intent.