Loading ToolBench
A plugin-based system for managing code, infrastructure, and spark projects with skills for planning, configuration, and domain checking
Arness exhibits severe definition quality gaps across all 8 tools. Tool names lack action verbs (arn-code-ensure-config, arn-infra-save-plan use hyphenated prefixes rather than verb_noun patterns). Descriptions vary wildly: some are adequate (arn-spark-naming-whois-check: 140 chars), others are vague or template-like (arn-spark-visual-strategy-capture references a 'template' without explaining what it generates). Parameter schemas are present for 3 tools but missing entirely for 5 tools (arn-code-ensure-config, arn-infra-ensure-config, arn-spark-ensure-config have no visible input schemas). No output schemas documented. Error handling is absent, scripts use bash exit codes and stderr messages, but no structured error responses or recovery guidance. No security annotations, no idempotency hints, no permission gates. The codebase is bash/JavaScript/Python scripts, not MCP tool definitions, tool metadata appears inferred from file paths and comments rather than explicit registration.
Validates cache at .arness/arn-code-ensure-config.local.json by checking fingerprints against current state (CLAUDE.md Arness section, agent models, templates, user profile, gitignore, GitHub labels)
Finds plan file and creates project structure with SOURCE_PLAN.md, plans/, and reports/ directories
Validates cache at .arness/arn-infra-ensure-config.local.json by checking fingerprints against current state (CLAUDE.md Arness section, agent models, templates, user profile, gitignore, GitHub labels)
Creates infrastructure plan project directory structure with SOURCE_PLAN.md, plans/, and reports/ directories
Validates cache at .arness/arn-spark-ensure-config.local.json by checking fingerprints against current state (CLAUDE.md Arness section, agent models, templates, user profile, gitignore, GitHub labels)
Domain availability checker using RDAP (primary) with WHOIS fallback. Reads JSON from stdin with domain list, outputs JSON with availability status for each domain
5 of 8 tools have no visible input schemas. arn-code-ensure-config, arn-infra-ensure-config, and arn-spark-ensure-config are cache-check scripts with no documented parameters, violating the requirement that every tool parameter have a type and description.
Tool names do not follow verb_noun convention. Names like 'arn-code-ensure-config' and 'arn-infra-save-plan' use hyphenated prefixes and lack clear action verbs. LLMs cannot infer intent from names alone. Should be 'validate_code_config', 'save_infra_plan', etc.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields without knowing what each tool returns. E.g., arn-code-save-plan creates directories but does not document what paths or metadata it returns.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 35 | 2026-07-28+ | v2 |
Domain availability checker using RDAP (primary) with WHOIS fallback (Python implementation). Reads JSON from stdin with domain list, outputs JSON with availability status for each domain
Visual test capture script template that generates project-specific capture scripts for taking screenshots of screens/routes at specified viewports in light and dark modes using Playwright
No error handling or recovery guidance. Scripts exit with bash codes and stderr messages (e.g., 'cache miss: no cache file'). LLMs receive raw exit codes with no actionable next steps. Should return structured errors with recovery hints.
Tool definitions appear inferred from file paths and comments, not explicitly registered. No visible MCP server registration code, tool manifest, or capability negotiation. Codebase is raw bash/JS/Python scripts, not an MCP server implementation.