Loading ToolBench
Score every source file in a repository on six architectural pain dimensions
Two tools with clear, domain-specific names (painpoints_file, painpoints_repo) and detailed descriptions (180 - 220 chars, within baseline 194 avg). Input schemas present with typed parameters and descriptions. However, output schemas are not documented in the visible code, LLMs cannot predict return structure. Parameter descriptions are good but lack explicit constraints (e.g., no enum for file types, no min/max for 'limit'). Error handling is mentioned in descriptions but recovery guidance is absent. No tool annotations (readOnlyHint/destructiveHint). Composition is sound: two focused tools, no overlapping responsibilities.
Score a single source file on six architectural pain dimensions and, when .painpoints/rules.json has matching model rules, on those project agent rules. Returns every score, the worst dimension, and findings: architecture dimensions at 2.0 or above plus rule:<id> violations. Reuses the saved report as a cache, so an unchanged file costs no tokens.
Score every source file in a repository on the six architecture dimensions and any matching agent-rule model questions from .painpoints/rules.json. Returns the distribution plus the worst files, each with its findings. Writes the full JSON and Markdown report next to the repository. Only files whose contents changed cost tokens.
Output schemas not documented. LLMs cannot predict return structure (fields, types, nesting). Descriptions mention 'returns the distribution plus the worst files' but no formal schema visible in code.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). painpoints_repo has WRITE risk but no destructiveHint declared. painpoints_file is READ_ONLY but no readOnlyHint. Agents cannot infer safety properties.
'limit' parameter lacks bounds. No min/max specified; LLMs could pass 0, negative, or absurdly large values. Baseline: numeric params should declare range (e.g., 1 - 10000).
Error handling lacks recovery guidance. Descriptions mention 'findings' and 'violations' but do not explain what to do if classification fails, API key is missing, or a file is unrecognized.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 61 | 2025-06-18+ | v2 |
'include' parameter (painpoints_repo) is an array but no description of format, max length, or path syntax. LLMs may pass invalid paths or too many entries.