Loading ToolBench
Versioned, provenance-grounded knowledge layer over a codebase, served via MCP
Three well-named, read-only tools with strong descriptions (194 - 280 chars, above baseline 194 avg). All tools have complete input schemas with typed parameters and descriptions. Output schemas are documented via return type annotations (FindProvenanceResult, GetKnowledgeResult, SearchKnowledgeResult). Tool names follow verb_noun pattern (find_, get_, search_). Annotations correctly declare readOnlyHint=True. Gaps: no error handling guidance (what to do if sha is invalid, embeddings missing, or query fails); no pagination/limit enforcement in search_knowledge despite potentially large result sets; token_budget parameter lacks min/max constraints in descriptions.
Knowledge grounded at ``file:line@sha``: the spans there + artifacts derived from them. ``sha`` defaults to the latest-ingested snapshot. Unknown location -> empty hits.
Knowledge units for a ``target`` (a logical key, file path, or module), ranked + trimmed. Resolution: exact logical key, else prefix, else by file, else by module. Ranked by confidence then kind then key; greedily trimmed to ``token_budget`` (>=1 unit kept), with the dropped count reported as ``omitted`` (no silent truncation).
Semantic search over grounded knowledge: embed ``query``, cosine-rank artifacts in the snapshot, return budget-trimmed grounded units (each with provenance/method/confidence/ freshness). ``sha`` defaults to the latest-ingested snapshot. Requires ``kb embed`` to have populated embeddings; otherwise returns empty.
No error handling guidance. Tools lack recovery hints for common failures (invalid sha, missing embeddings, empty results). LLMs cannot self-correct without actionable error messages.
token_budget parameter lacks min/max constraints in description. Rubric requires 'minimum and maximum for numeric parameters'. Current description only states 'default: 2000, minimum 1 unit kept' but does not specify upper bound or validation behavior.
search_knowledge may return unbounded results. Parameter 'k' (default 8) controls top-k, but no explicit cap on total response size or guidance on pagination. Large semantic search result sets could exhaust context windows.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 65 | 2026-07-28+ | v2 |
sha parameter description does not explain resolution behavior clearly. Users may not understand that 'defaults to latest-ingested snapshot' means the tool silently picks a version if sha is omitted, which could cause confusion in multi-version workflows.