Loading ToolBench
ShellWatch defines 8 tools with consistent naming (verb_noun pattern: create_session, send_keys, read_output, close_session, list_endpoints, get_endpoint, list_keys, get_key). All tools have descriptions (avg ~60 chars, baseline 194 chars) and input schemas with typed parameters. However, descriptions are terse and lack context about when to use each tool, prerequisites, or what happens on error. Parameter descriptions are minimal (e.g., 'The session ID to send keys to' lacks format/constraint details). Output schemas are not documented, LLMs cannot predict response structure. No error handling guidance. Risk annotations (WRITE/READ_ONLY) are present but not exposed in tool definitions. Missing idempotency guidance for stateful operations like create_session.
Close an active terminal session
Create a terminal session to a remote SSH endpoint
Get details about a specific SSH endpoint
Get details about a specific SSH key
List all available SSH endpoints
List all available SSH keys
Read terminal output from an active session, optionally from a specific offset
Output schemas not documented. LLMs cannot predict response structure (fields, types, pagination). Forces agents to guess what data is available after each call.
Descriptions are terse (avg 60 chars vs baseline 194 chars) and lack context. Missing: when to use each tool, prerequisites, what happens on error, whether operations are idempotent/retryable.
Parameter descriptions lack format/constraint details. E.g., 'keys' array in send_keys lacks examples of valid key commands (text:, enter, ctrl+c). 'afterOffset' lacks range (0 - max_output_size?).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 48 | 2026-07-28+ | v2 |
Send keyboard input to an active terminal session
No error handling guidance. Tools return errors but descriptions do not explain recovery steps (e.g., 'Session not found, call list_endpoints first to verify endpoint exists').
Risk annotations (WRITE/READ_ONLY) are metadata but not exposed in tool definitions. LLMs cannot see which tools are destructive and should be confirmed before execution.