Loading ToolBench
A skill catalog management and routing system for Claude Code. Manages SKILL.md files across local skills and plugins, builds a searchable skill index, and provides routing/intake/auditing infrastructure.
Claude Orchestra exposes 4 tools with critical definition gaps. Tool names lack action verbs (skill-index-refresh, build-skill-index, orchestra-route, orchestra-installer are noun-heavy or unclear). Descriptions exist but are verbose (200-400+ chars) and lack LLM-optimized structure. Input schemas are either missing entirely or minimally specified (only 'flags' string with no enum constraints). No output schemas documented. Error handling and recovery guidance absent. Tools appear to be shell/Python scripts with inferred registration rather than explicit MCP tool definitions, cannot verify actual schema compliance from source.
Single source of truth for the Claude Code skill catalog. Scans all SKILL.md files (local ~/.claude/skills/ + plugin ~/.claude/plugins/), writes compact discovery stubs, registers/updates the qmd 'skills' collection, regenerates ~/.claude/.orchestra-scan.md, and writes a signature file for freshness detection.
Claude Orchestra installer. Installs the orchestra quintet (orchestra-router, orchestra-intake, hallucination-guard, skill-selector, auditor agent), routing hook, constitution template, and scans ~/.claude/ to write a manifest for personal classification. Idempotent and backs up all files before mutation.
UserPromptSubmit routing hook (v3, lean). Injects a one-line routing reminder each turn. Routes silently via skill retrieval (skill-selector; qmd skills collection if installed). Announces only delegation. Handles capability gaps and recommends protocol. Files new installs with orchestra-intake.
SessionStart hook that cheaply checks whether the skill catalog has changed since the last build. If yes, rebuilds the skill index in the background (never blocks session start). If no, exits immediately with no cost.
No action verbs in tool names. 'skill-index-refresh', 'build-skill-index', 'orchestra-route', 'orchestra-installer' are noun-heavy or unclear. LLMs cannot infer intent from names alone. Should be: 'refresh_skill_index', 'build_skill_catalog', 'route_user_prompt', 'install_orchestra'.
Input schemas missing or incomplete. 'build-skill-index' and 'orchestra-installer' expose only a 'flags' string parameter with no enum constraints, type validation, or format specification. LLMs cannot distinguish valid flag combinations. Should declare flags as enum or separate boolean parameters.
No output schemas documented. Tools return unstructured results (file paths, counts, status messages). LLMs cannot plan downstream calls or extract required fields. Must document return types: e.g., build-skill-index returns {stub_count: int, unique_skills: int, scan_file: string}.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 21 | 2026-07-28+ | v2 |
Descriptions are verbose (200-400+ chars) and lack LLM-optimized structure. 'orchestra-route' description is 300+ chars with embedded implementation details ('qmd skills collection', 'skill-selector'). Should be 50-150 chars: 'Routes user prompts to relevant skills via semantic search. Announces delegation only.'
No error handling or recovery guidance. Tools fail silently or with generic messages (e.g., 'Build script not found'). No guidance on retryability, user-fixable errors, or next steps. Should return: {status: 'error', code: 'SCRIPT_NOT_FOUND', message: 'Build script missing. Run orchestra-installer --guided to set up.', retryable: false}.