Loading ToolBench
Interactive profiling + harness compiler + evolution engine. Generates a personalized AI agent harness from a user's tacit knowledge, genre, and goals. Exposes BYOH capabilities as a stdio MCP server.
BYOH defines 8 tools with consistent naming (verb_noun pattern: profile_*, build, author_skill, install_plugin) and reasonable descriptions (avg ~100 chars). All tools have input schemas with typed parameters and descriptions. However, output schemas are not documented in the visible code, responses are inferred from implementation. Parameter descriptions lack detail on constraints, formats, and error recovery. No tool annotations (readOnlyHint/destructiveHint) despite clear risk classifications. Error handling is basic; no guidance on retryability or user-fixable vs fatal errors. Composition is sound (single responsibility per tool), but descriptions could be more LLM-optimized per pattern:tool-description baseline (target 50-200 chars).
Author/override a skill body (markdown with YAML frontmatter). Safety-gate skills (critic, seesaw, stagnation) are rejected.
Synthesize and compile a harness bundle from a Confirmed profile. Async (spawn_blocking for dry_run).
Render and install a synthesized harness plugin to a target host (all, claude, vscode, etc.).
Confirm a profile (transition from Draft to Confirmed). Requires genre and optional goal_30d.
Create a draft profile (and optionally autoscan paths). Returns the created profile.
Run the interview questionnaire (rule-based, no LLM). Returns updated profile.
Read a BYOH user profile by slug. Returns the full profile as JSON.
Output schemas not documented. Tool descriptions state what is returned (e.g., 'Returns the full profile as JSON', 'Returns updated profile') but do not specify field names, types, or structure. LLMs cannot plan downstream tool calls or extract required data without knowing response shape.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite explicit risk classifications in metadata. Tools marked WRITE (profile_create, profile_interview, profile_confirm, author_skill, install_plugin) should declare destructiveHint=true; READ_ONLY tools should declare readOnlyHint=true. This enables agents to reason about side effects and retry safety.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | <=2025-11-25 | v2 |
Scan filesystem paths for evidence (async, spawn_blocking). Returns updated profile.
Parameter descriptions lack constraint details. 'slug' is described as 'Profile identifier (slug)' but does not specify format (alphanumeric, length limits, forbidden characters). Tests show rejection of '/', '..', uppercase, these constraints must be in descriptions. 'language' accepts 'en', 'ko' but no enum or format guidance. 'genre' in profile_confirm accepts 'developer, creator, researcher, business' but is not declared as an enum.
Error handling lacks recovery guidance. Tests show errors (e.g., 'build must refuse a Draft profile', 'hostile_slugs_are_rejected') but tool descriptions do not explain what to do on failure. No guidance on retryability, user-fixable errors, or next steps. Per pattern:recovery-guide, errors should tell LLMs what to do next.
install_plugin parameter 'target' is described as 'Target host (all, claude, vscode, etc.)' but lacks enum constraint. 'scope' is optional with no description of valid values or when to use it. 'host' boolean parameter is ambiguous, does it mean 'install to host machine' or 'is this a host-specific config'? Descriptions should clarify intent and constraints.