Loading ToolBench
Provides tools to run Orca and Snyk security scans.
Server has 3 tools with complete input/output schemas and descriptions. Naming follows verb_noun pattern (scan_*, get_*). However, descriptions are generic and lack LLM-optimized guidance on when to use each tool vs. alternatives. Parameter descriptions are present but minimal (10-30 chars). No error handling guidance, no recovery paths, no actionable error messages documented. Output schemas exist but lack field-level descriptions. Missing tool annotations (readOnlyHint, idempotentHint). Security concern: project_path parameter accepts arbitrary paths with no documented validation or traversal protection.
Retrieve the list of Snyk organizations accessible via the configured Snyk API token.
Run an Orca security scan on a specified project path.
Run a Snyk security scan on a specified project path.
Descriptions lack LLM-optimized guidance. 'Run an Orca security scan' (34 chars) does not explain WHEN to use scan_orca vs scan_snyk, what prerequisites exist (CLI installed?), or what the output contains. Baseline: 194 chars avg, 50-200 chars optimal.
project_path parameter accepts arbitrary filesystem paths with no documented validation, length limits, or traversal protection. LLM could pass '../../etc/passwd' or extremely long paths. No mention of allowed directories or security constraints.
No error handling guidance. Tool descriptions do not explain what errors are possible, whether they are retryable, or what the LLM should do next. E.g., 'CLI not installed' vs 'invalid path' vs 'API auth failed' all need distinct recovery paths.
Inferred effective spec: 2025-06-18+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 52 | 2025-06-18+ | v2 |
Output schema fields lack descriptions. 'results' is documented as 'Scan results (JSON object from Orca CLI or error details)' but does not specify structure, field names, or how to extract vulnerabilities. LLM cannot reliably parse output.
No tool annotations. Tools lack readOnlyHint (all are read-only), idempotentHint (scans are idempotent), or destructiveHint. LLM cannot infer safety properties and may over-retry or under-retry.