Loading ToolBench
Single tool 'run-collection' has a clear verb-noun name and basic schema with required/optional parameters. However, the tool description is generic (47 chars, below the 50-200 char LLM-optimized range), and parameter descriptions lack actionable detail about constraints, formats, and error recovery. No output schema is documented. Error handling is minimal, the code shows basic try-catch but no guidance for the LLM on what to do if Newman fails. The tool is READ_ONLY (safe), but lacks the depth expected of production-grade tools.
Run a Postman Collection using Newman
Tool description is too short (47 chars) and lacks context. Should explain WHEN to use this tool, WHAT it returns, and any prerequisites (e.g., 'Newman must be installed'). Current description 'Run a Postman Collection using Newman' does not guide LLM selection or error recovery.
Parameter descriptions lack actionable constraints. 'iterationCount' says 'Optional number of iterations to run' but does not specify min/max bounds, default value, or what happens if omitted. LLMs cannot infer valid ranges and may pass invalid values.
No output schema documented. The code returns text content but does not specify what fields the LLM should expect (e.g., test results, pass/fail counts, error details). LLMs cannot plan downstream actions without knowing the response structure.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 47 | 2026-07-28+ | v2 |
Error handling is minimal. The code catches errors but returns generic text responses. No guidance for the LLM on whether errors are retryable, what caused them, or what to do next. E.g., 'Collection not found. Try providing a valid file path or URL.'
Parameter 'collection' accepts both paths and URLs but the description does not clarify the format or provide examples of valid inputs. LLMs may pass invalid paths or malformed URLs.