Loading ToolBench
Plumbline has 8 well-named tools with clear, action-oriented names (plumbline_context, plumbline_query, plumbline_verify, etc.). Descriptions are detailed and contextual (avg ~180 chars), explaining WHEN to call each tool. However, input schemas lack explicit type declarations and constraints. Parameters like 'phase' in plumbline_advance accept free-form strings instead of enums. Output schemas are entirely undocumented, LLMs cannot predict return structure. Error handling is absent from tool definitions. No tool annotations (readOnlyHint/destructiveHint) despite clear risk classifications in metadata. Composition is strong: tools chain logically (context → query → verify → explain). Parameter naming is consistent (project_root, file_path, plan_id). Defaults are sensible (limit=20, limit=50, plan_id='').
Advance a task through a phase gate with evidence. Each phase requires specific evidence. The gate validates your evidence is real (not bare assertions like 'yes' or 'ok'). Returns success with next phase instructions, or rejection with what's missing. Phases: CHECK → CODE → TEST → GRADE → COMMIT → ADVANCE
Find files that change together with this file. Call before editing a file. If auth.py and auth_test.py always change together, you should read auth_test.py before editing auth.py. Prevents the 'fixed the function, forgot the test' mistake.
Get project intelligence at session start. Returns conventions, pitfalls, architectural decisions, hot files, and co-change patterns — everything an AI needs to write project-consistent code. Call this first.
Explain a rejection in detail. After plumbline_check returns BLOCK or plumbline_verify returns REJECT, call this with the finding ID to understand: what was detected, why it matters, what the fix should look like, and what the confidence score means.
Start tracking a development plan. Load a markdown plan file, parse tasks, and begin tracking state. Use this when starting multi-step work (3+ tasks). The plan keeps you honest — evidence gates prevent quality decay on later tasks.
No output schemas documented. LLMs cannot predict return structure (fields, types, pagination). All 8 tools return bare strings; no structured response definition.
plumbline_advance 'phase' parameter accepts free-form strings instead of enum. Should constrain to: CHECK, CODE, TEST, GRADE, COMMIT, ADVANCE. Prevents LLM hallucination of invalid phases.
No tool annotations despite clear risk metadata. plumbline_init and plumbline_advance are WRITE operations; plumbline_verify is READ_ONLY. Missing readOnlyHint/destructiveHint annotations prevent LLMs from reasoning about side effects.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 56 | <=2025-11-25 | v2 |
Search project knowledge mid-task. Free-text search across conventions, decisions, pitfalls, and patterns. Use when you need to look something up: 'Has this pattern been tried before?' 'What's the convention for error handling here?'
Check plan progress and current state. Returns task states, active phase, and completion percentage. If plan_id is omitted, returns the most recently created plan.
Deep pre-commit verification. Run before committing. Runs targeted mutation testing on changed lines and spec compliance checks. Returns ACCEPT, ACCEPT_WITH_WARNINGS, REJECT, or PARTIAL. Target: <30s. Surviving mutants are reported as questions: 'Your code still passes tests if X is changed to Y. Is that intentional?' If REJECT: do not commit. Read the findings. Fix first.
No error handling guidance in tool definitions. plumbline_verify returns REJECT/ACCEPT/PARTIAL but no description of what LLM should do on each outcome. plumbline_explain references 'finding_id' but no guidance if ID is invalid.
plumbline_advance 'evidence' parameter is untyped dict with no schema. What fields does each phase expect? No validation rules documented. LLMs cannot construct valid evidence payloads.