Loading ToolBench
MCP server — persistent experiment tracking and semantic code search for AI coding agents
claude-labbook demonstrates solid tool design with clear naming conventions (verb_noun pattern), comprehensive parameter schemas, and well-structured descriptions. All 12 tools have explicit input schemas with typed properties and descriptions. However, output schemas are not documented, error handling guidance is minimal, and several tools lack actionable recovery instructions. The domain is specialized (experiment tracking) and well-suited to the tool set, but production readiness is limited by STDIO transport and missing error classification patterns.
Before modifying a component, check what changes have already been tried and their outcomes. Also searches for semantically similar trials from other sessions. ALWAYS call this before reverting or significantly changing code in a component that may have prior trial history.
Get a compact summary of all active sessions, recent trials, environment facts, and key decisions for this project. Call this at the start of any task, after compaction, or when context seems incomplete. This is the primary context recovery tool.
Retrieve all environment facts, optionally filtered by category.
Follow the chain of trials from a starting trial to see the full evolution of attempts.
List all sessions, optionally filtered by status.
Record an architectural or strategic decision. Decisions are broader than trials — they inform multiple trials (e.g., 'Using httpx instead of requests for async support').
Output schemas not documented. LLMs cannot plan downstream tool calls or extract required fields without knowing response structure. All 12 tools lack documented return types.
Error handling lacks recovery guidance. Tools do not categorize errors as retryable, user-fixable, or fatal. No actionable error messages guide LLM recovery (e.g., 'Session not found. Call list_sessions() to see active sessions.').
No pagination or result limits documented for list_sessions and get_briefing. Large result sets could exhaust context window. Missing limit parameters and total_count fields in response documentation.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 63 | 2026-07-28+ | v2 |
Record a machine-specific or environment detail that Claude needs to remember across compactions. Examples: 'python_cmd=python (NOT python3)', 'deploy_cmd=railway up --service relay'.
Record the outcome of a code change that was tested. Call this AFTER testing any meaningful change. This is the most important tool — it prevents repeating failed approaches after compaction.
Mark a session as resolved when the problem is solved. Record what ultimately worked.
Index or re-index the project codebase for semantic search. On first run, scans all files. On subsequent runs, only re-indexes files whose content has changed (incremental, based on content hash). Call this at session start or when significant new files have been added.
Semantic search across the indexed codebase. Use this to find code by what it does, not just by keyword. For example: 'token refresh logic', 'database connection pooling', 'error handling middleware'. Also useful for humans asking questions about the codebase.
Start a new experiment session for a problem or feature you're working on. Sessions group related trials together.
Destructive operations (resolve_session, log_trial with 'reverted' outcome) lack confirmation or dry-run patterns. Agents could accidentally mark sessions resolved or revert trials without verification.
Parameter 'component' in log_trial and check_before_change accepts free-form strings with no validation guidance. Should document expected format (file path, function name, module) and provide examples to prevent LLM hallucination.