Loading ToolBench
I Know KungFu — MCP server: the librarian (knowledge store), the verifier (Docker truth oracle), and the bench (learning-curve measurement). Never calls an LLM; all judgment stays with Claude.
Strong tool definitions with excellent descriptions (50-200 char range, LLM-optimized). All 6 tools have clear, actionable docstrings explaining WHEN to call them and what they enforce (honesty contracts, verification requirements). Input schemas present with types and descriptions. However, output schemas are not formally documented in the visible code, only inferred from docstrings. Parameter descriptions are comprehensive but some optional params lack clarity on defaults/constraints. No tool annotations (readOnlyHint/destructiveHint) despite clear risk profiles (WRITE vs READ_ONLY). Error handling guidance is implicit in descriptions but not structured.
Measure competence on the held-out suite. Proves the agent actually learns. Actions: - ``list``: show train/test task ids and their prompts. - ``selfcheck``: run the reference solutions (a healthy suite + sandbox passes all). - ``score``: verify your ``solutions`` ({task_id: {relative_path: content}}, or {task_id: {"files": {...}, "iters": n}}) against hidden tests; pass ``label`` ('cold' or 'warm') to persist the run. - ``report``: render the cold-vs-warm learning curve from saved runs. Never fakes numbers: if Docker is down, runs are marked not measurable.
Record a mistake as a misconception -> correct-model lesson. Capture the *reasoning* error in ``misconception``/``trigger`` (when this wrong thought occurs), the bad code in ``wrong``, the fix in ``right``, and the structural reason in ``model``. The lesson is keyed by ``trigger`` so it surfaces next time that reasoning recurs. Prefer ``source='verify-failure'`` with ``verified=True`` when the fix actually compiled — that is the strongest kind of lesson.
Manifest-first retrieval. Call BEFORE writing any code in a managed language. Returns the language skeleton (incl. negative knowledge), the lessons whose flawed reasoning matches your query, and only the matched cards — never the whole language. Read ``note``: if it says there is no grounded knowledge, everything you write is a HYPOTHESIS until you fetch real docs or verify it.
Persist distilled knowledge with provenance. YOU distil; the server stores. ``kind`` is one of ``syntax`` / ``stdlib`` / ``library``. Provide a real ``source`` (the URL you fetched) so the card is GROUNDED rather than a HYPOTHESIS. Set ``verified=True`` only if it actually compiled in the sandbox.
Output schemas not formally documented. Tool descriptions infer structure (e.g., 'returns dict with saved, slug, kind, ep...') but no JSON Schema definition visible. LLMs cannot reliably parse unstructured responses or plan downstream calls.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear risk profiles. kungfu_save_card and kungfu_learn are WRITE operations; others are READ_ONLY. Annotations enable agents to reason about side effects and retry safety.
Optional parameters (query, kind, limit in kungfu_lookup; title, source, verified in kungfu_save_card) lack explicit default behavior documentation. LLMs cannot infer whether omitting 'limit' returns 6 items or all items.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | <=2025-11-25 | v2 |
Orientation. Call this FIRST when about to work in a language. Reports what the knowledge base knows, whether the Docker verifier is available, lesson counts, and the last benchmark. If ``docker_available`` is false, you cannot prove code works — say so honestly instead of claiming it.
Compile/type-check/test code in the Docker sandbox. The truth oracle. ``files`` maps relative paths to contents and must form whatever the language image expects (e.g. a minimal Gleam project: ``gleam.toml`` + ``src/*.gleam``). Honesty rule: if ``available`` is false you may NOT claim the code works; tell the user it is unverified. If ``ok`` is false, fix it, then record the wrong->right with ``kungfu_learn``.
Error handling is implicit in descriptions ('if docker_available is false, say so honestly') but not structured. No explicit error classification (retryable, user-fixable, fatal) or recovery guidance in tool contracts.
Parameter 'action' in kungfu_bench accepts free-form string ('list', 'selfcheck', 'score', 'report') but is not declared as an enum. LLMs may hallucinate invalid actions.