Loading ToolBench
Single tool with minimal quality. Tool name is action-oriented (get_allure_report), but description is generic (11 words, 45 chars). Input schema is present with proper type and description for the single parameter. However, tool lacks output schema documentation, no error handling guidance, no parameter validation hints, and the exception handling returns raw error strings rather than structured recovery guidance. The implementation catches all exceptions and returns str(e), which provides no actionable recovery path for the LLM. No pagination, no structured response shape documented. Severity of issues: output schema completely missing (critical), error handling non-existent (critical), description too minimal (high).
read allure report and return json data
No output schema documented. Tool returns a JSON string via json.dumps(), but the LLM has no visibility into the structure of that JSON (test-suites, test-cases, steps, etc.). LLMs cannot plan downstream usage without knowing what fields to expect.
Error handling is non-recoverable. Exception handler returns raw str(e) with no guidance to the LLM on what to do next. A FileNotFoundError mentioning 'suites.json' gives no hint that the user should provide a valid Allure report directory or check the path. Should return structured error with recovery suggestions (e.g., 'Directory not found. Ensure results_dir points to an Allure report with data/suites.json.').
Description too short and generic (45 chars: 'read allure report and return json data'). Does not explain WHEN to use this tool, WHAT format the results_dir should be, or WHAT the output structure contains. Should be 100-200 chars explaining the purpose, prerequisites, and output shape.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 45 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 38 | - | v1 |
Parameter 'results_dir' description is minimal ('Path to the allure report directory'). Does not clarify expected structure (must contain data/suites.json and data/test-cases/), or examples of valid paths. Should guide LLM on what constitutes a valid Allure report directory.
No input validation or constraint hints. Parameter results_dir accepts any string; code relies on downstream file system checks. Should include format guidance in description (e.g., 'Absolute or relative path. Must point to directory containing data/suites.json.').
Response is unstructured JSON string, not a typed object. Tool returns json.dumps(...) as a plain string. LLMs must re-parse JSON from the string, increasing error and token overhead. Should return the parsed object directly with a documented schema declaring the shape: {test-suites: [{name, description, status, start, stop, test-cases: [...]}, ...]}.