Loading ToolBench
A human-curated knowledge base server for data work, exposing tools over MCP (Model Context Protocol) for searching, retrieving, and managing concepts, metrics, and related knowledge. Runs no SQL and no LLM.
ochakai demonstrates solid tool design with clear verb_object naming (search_concepts, list_concepts, get_concept, put_concept, report_outcome, verify_concept, delete_concept). All 7 tools have descriptions (avg ~120 chars, within 10-1024 range). Input schemas are present with typed parameters and descriptions. However, output schemas are not documented in the visible code, LLMs cannot predict response structure. Error handling descriptions are minimal; tools lack recovery guidance (e.g., what to do if a concept is not found). Parameter constraints are partially specified (e.g., limit ranges stated in descriptions but not as JSON Schema min/max). Risk annotations (READ_ONLY, WRITE, DESTRUCTIVE) are present but not formalized as toolAnnotations in the MCP protocol.
Delete a concept by marking it as a tombstone. Prevents recreation of the same ID.
Retrieve a single concept by its ID. Returns the concept document, metadata, and linked_from (concepts pointing at it).
List concepts from a feed (e.g., recent, stale, unverified) with optional filters (types, statuses, trusts, tags, source, prefixes). Returns paginated results with limit (default 100, max 1000) and cursor-based navigation.
Create a new concept or replace an existing one. Takes a concept ID and YAML document. Creates on a free ID, replaces on a taken one. Refuses to revive a curated tombstone or replace a verified entry without report_outcome.
Record an outcome (e.g., failed, succeeded, inconclusive) for a concept with an optional note. Used to report whether a concept's guidance worked in practice. Returns outcome totals, not the individual reports.
Search for concepts by query string with optional filters (types, statuses, trusts, tags, source, prefixes). Returns paginated results with limit (default 10, max 50) and cursor-based navigation.
Output schemas not documented. LLMs cannot predict response structure (fields, types, pagination format). Tools return paginated results but response schema is invisible in code.
Error handling lacks recovery guidance. No descriptions of what errors are possible, whether they are retryable, or what the LLM should do next (e.g., 'concept not found, try search_concepts first').
Parameter constraints (limit ranges, enum values for outcome types) are stated in descriptions but not formalized in JSON Schema (no min/max, no enum arrays). LLMs cannot parse constraints from text alone.
Inferred effective spec: 2026-07-28+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | B | 70 | 2026-07-28+ | v2 |
Mark a concept as verified by a human curator. Locks the concept against further replacement until a new report_outcome is filed.
Risk annotations (READ_ONLY, WRITE, DESTRUCTIVE) are present in code comments but not exposed as MCP toolAnnotations (readOnlyHint, destructiveHint). Protocol does not declare which tools are safe to retry.
put_concept description mentions 'Refuses to revive a curated tombstone or replace a verified entry without report_outcome' but does not explain what error is returned or how to recover. Conditional behavior is underdocumented.