Loading ToolBench
Offline-first RAG server with hybrid semantic+keyword retrieval. Retrieval-only, no LLM sampling capability. Exposes hybrid_search tool via JSON-RPC over stdio.
CyClaw exposes 11 tools with significant quality gaps. Most tool descriptions are present but generic (under 100 chars, no guidance on when/why to use them). Input schemas exist for most tools but lack rigor: parameters often have no descriptions, no validation constraints, and no type information beyond the minimal JSON Schema. The `hybrid_search` tool has the strongest schema (enums, defaults, ranges); others like `repo_context_read` and `read_surface_manifest` have empty input objects with no guidance. Output schemas are entirely undocumented, LLMs cannot plan downstream calls or extract needed fields. Error handling is absent from all tool definitions. The server exhibits a pattern of workspace-scoped read/write operations (local_repo_read, proposal_workspace_write_current, etc.) which are well-intentioned for sandboxing but lack clear descriptions of what "workspace boundary" and "scoped task" mean to an LLM. Security is partially addressed (read vs. write distinction, no credentials in params) but not explicitly declared in tool metadata. Composition is weak: multiple similar tools (local_repo_read vs. repo_context_read vs. read_file) create confusion about when to use each. No batch operations, no pagination guidance, no chaining IDs in responses.
Write proposal.md through the explicit proposer workspace boundary
Search local .md corpus using semantic + keyword retrieval with RRF fusion
List readable workspace entries without exposing holdout contents
Read one visible file from the scoped workspace
Write a candidate file below current/ after human approval
Run the injected read-only RAG lookup for the scoped task
Read a visible workspace file
Multiple tools read files from workspace with unclear distinctions: repo_context_read vs. local_repo_read vs. read_file all appear to read workspace contents, but descriptions do not explain when to use each. LLMs will conflate them and make suboptimal selections.
Empty input schemas for 5 tools (repo_context_read, read_surface_manifest, read_train_failures): these accept no parameters yet provide no description of what they return or when to call them. Score: 0/100 for schema. Blocks LLM planning.
No parameter descriptions on any input schema. All 7 tools with parameters lack descriptions of what each parameter does, valid ranges, constraints, or format. E.g., 'target' in local_repo_read has no guidance on path format, whether absolute or relative, or what happens if file does not exist.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 44 | 2026-07-28+ | v2 |
Read the local surface manifest
Read visible train artifacts
Read the runner-provided surface manifest for this scoped workspace
Write a candidate file under current/ only
No output schemas documented for any tool. LLMs cannot plan chains: after calling hybrid_search, what fields are returned? Can the result be passed to another tool? What does repo_context_read return, a file object, raw text, metadata? Undocumented responses force guessing and degrade agent reasoning.
Workspace-scoped semantics (scoped workspace, current/, holdout contents, proposer workspace boundary) are mentioned in descriptions but never defined. An LLM reading 'Write proposal.md through the explicit proposer workspace boundary' has no idea what that means or what the expected outcome is.
No error handling guidance. What happens if local_repo_read is called on a non-existent file? If write_current_file tries to write outside the current/ directory? If rag_search_readonly finds no matches? Tools provide no recovery hints, forcing LLMs to fail silently or retry blindly.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). The Risk column in the spec identifies READ_ONLY vs WRITE tools, but this is not exposed in the schema or as tool metadata. LLMs cannot distinguish safe retry paths from operations with side effects.
Parameter naming inconsistencies: 'target' is used in local_repo_read, list_workspace, read_file, and write_current_file but means different things (file path, directory path, etc.). Should be 'file_path', 'directory_path', or similar to disambiguate.