Loading ToolBench
The open format for live, evidence-backed datasets — protocol, CLI, and curation MCP server.
Frame demonstrates solid tool design with complete schemas, clear descriptions, and thoughtful composition. All 9 tools have explicit input schemas with proper JSON Schema structure and type definitions. Descriptions are substantive (100-250 chars typical), explaining WHAT the tool does and WHEN to use it. Tool naming follows verb_noun convention (add_entity, set_fact, query). Composition is excellent: tools chain naturally (add_entity → set_fact), batch variants exist (set_facts, add_entity_with_facts), and output includes necessary IDs for downstream calls. Error handling is present but minimal, no recovery guidance or actionable error messages visible in source. Parameter descriptions are present but sparse; many lack format/constraint details. Output schemas are not explicitly documented in the source code.
Create a new entity. Required before set_fact on it. Returns the entity_id.
Combine entity creation and bulk fact-set in one call. Highest-throughput for the common pattern: one page read → one entity with N fields. Returns entity_id and all fact_ids.
Add an additional source to an existing fact without changing the value.
Mark a previously-set fact as no longer trusted. Reverts the (entity, field) to the most recent prior non-deprecated fact, or unsets.
Force regeneration of the SQLite index and rows.ndjson from events.ndjson. Idempotent.
Read the current state of the frame. mode=all returns every row; mode=entity returns one; mode=field filters by a field; mode=sql runs a read-only SQL query against the SQLite index. Set include_sources=true on non-sql modes to attach each field's primary source. For full evidence including corroborating sources, query the `all_sources` view via mode=sql.
Output schemas not documented in source code. LLMs cannot plan downstream calls or extract required fields without knowing what query(), set_fact(), and other tools return. Baseline: 100% of A+ tools document return types.
Parameter descriptions lack format/constraint details. E.g., 'entity_id' says 'Optional. Slug-shaped (a-z0-9_-)' but no regex pattern or length limit. 'confidence' has min/max in schema but description is missing. LLMs cannot read JSON Schema constraints, they rely on description text.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | 2026-07-28+ | v2 |
Remove an entity from the rows projection. The entity's history remains in events.ndjson.
Set a field's value on an entity with required source. Last-write-wins by ts for (entity_id, field). Returns the fact_id.
Bulk-set multiple facts on one entity sharing a single source. Use when you've read a page and want to extract N fields from it — one call instead of N. Atomic: either every fact lands or none do.
Error handling and recovery guidance absent from source. No indication of what errors are retryable, user-fixable, or fatal. No actionable error messages visible (e.g., 'Invalid entity_id: must match [a-z0-9_-]'). Baseline: A+ tools include recovery guidance.
query() tool is overloaded with 4 modes (all, entity, field, sql). While documented, this violates single-responsibility principle. Consider splitting into query_all(), query_entity(), query_field(), query_sql() for clarity.
project() tool has minimal description ('Force regeneration of the SQLite index and rows.ndjson from events.ndjson. Idempotent.'). No explanation of WHEN to call it, what triggers the need, or what happens to in-flight queries. Baseline: 194 chars average for tool descriptions.