Loading ToolBench
libmake exposes 7 tools with basic descriptions and parameter schemas. All tools have descriptions (10-100 chars) and input schemas with type definitions. However, descriptions are minimal (avg 65 chars vs baseline 194), lack WHEN/WHY context, and omit actionable error guidance. Parameters lack enums where appropriate (e.g., makefile paths). Output schemas are undocumented, LLMs cannot predict response structure. No tool annotations (readOnlyHint/destructiveHint) despite clear risk levels. Error handling returns JSON but lacks recovery guidance. Naming is clear (verb_noun pattern) and single-responsibility. Composition is sound, tools chain naturally (list_targets → show_target → build). Missing: pagination, result limits, field descriptions for complex outputs.
Trigger a build for the given target and return structured results.
Run 'libmake clean' to remove build artifacts.
Explain what a build would do without executing anything.
Emit the provider graph as a Makefile.
List all build targets with their dependency and command counts.
Show full detail for a single build target: commands, deps, and file mtime.
Generate a Mermaid diagram of the build graph (or a subgraph rooted at target).
Output schemas undocumented. LLMs cannot predict response structure (e.g., list_targets returns array of {name, num_deps, num_commands}, but this is invisible to the agent). Agents must guess field names, risking extraction errors.
Descriptions too brief (avg 65 chars vs baseline 194). Missing WHEN/WHY context. E.g., 'build' says 'Trigger a build' but not 'Use after modifying source files' or 'Returns exit_code, stdout, stderr, duration_seconds'. Agents cannot determine when to call vs alternatives.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). 'clean' is destructive and 'build' is write-only, but agents have no machine-readable signal. Agents may retry destructive ops or assume read-only safety incorrectly.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 53 | 2026-07-28+ | v2 |
Error handling lacks recovery guidance. E.g., build() returns JSON with exit_code/stderr but no hint like 'If compilation fails, check src/main.c syntax' or 'Retry after fixing dependencies'. Agents cannot self-correct.
Parameter 'makefile' accepts free-form string paths with no validation, format hint, or enum. Agents may pass invalid paths (e.g., 'Makefile.txt' vs 'Makefile'). Should document expected format or provide discovery tool.