Loading ToolBench
Server has 9 tools with explicit schemas and descriptions, but quality is inconsistent. Naming lacks action verbs (e.g., 'arc.api.authentication' instead of 'authenticate_user'). Descriptions are present but generic (avg ~100 chars, below 194-char baseline). Parameter descriptions exist but lack actionable constraints (e.g., no format hints for 'projectPath'). Output schemas are completely undocumented, LLMs cannot plan downstream calls. Error handling is absent from visible code. Security concerns: credentials exposed as parameters (arc.api.authentication accepts username/password directly). No pagination support despite tools returning lists. Tool composition is poor: arc.deployment.deploy and arc.deployment.infrastructure overlap significantly.
Manage authentication and authorization for ARC applications
Send notifications across multiple channels in ARC applications
Configure CI/CD pipelines for ARC applications
Deploy an ARC application to a specified environment
Generate infrastructure as code for deploying ARC applications to various cloud platforms
Search the ARC documentation for specific topics, concepts, or components
Credentials exposed as tool parameters: arc.api.authentication accepts username and password directly. This violates secret-injection pattern, credentials will be logged in agent traces and prompt history.
No output schemas documented. LLMs cannot infer what fields are returned, preventing downstream tool chaining and forcing discovery detours.
Tool names lack action verbs. 'arc.api.authentication' should be 'authenticate_user' or 'login_user'. 'arc.deployment.infrastructure' should be 'generate_infrastructure'. LLMs infer intent from names, generic names cause misselection.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 54 | 2026-07-28+ | v2 |
Generate a new controller for an existing ARC project
Generate scaffolding for a new ARC microservice including models, controllers, and repositories
Generate a new data model for an existing ARC project
Parameter descriptions lack actionable constraints. 'projectPath' has no format hint (absolute vs relative, must exist?). 'region' has no enum or examples. LLMs cannot validate inputs without explicit constraints.
No error handling or recovery guidance in visible code. Tools return success/failure with no actionable next steps for LLMs. No distinction between retryable and fatal errors.