Loading ToolBench
Four tools with basic descriptions and minimal schemas. Tool names follow verb_noun convention (get_*, query_*), which is good. However, descriptions lack actionable detail for LLM selection, they don't explain WHEN to use each tool or dependencies between them. Parameter descriptions are present but generic. No output schemas documented. Error handling is absent, no guidance on retryability, user-fixable errors, or recovery paths. The server exposes external API calls (requests.post/get) without timeout, retry, or error classification logic.
Get all supported ecosystems. An ecosystem is a package manager or repository for a specific language. Returns: Dictionary of ecosystems with the key being the ecosystem name and the value being the programming language / OS. To use the ecosystem name in the query_package_cve function, you must use the exact ecosystem name as it is in the dictionary.
Query the OSV database for a CVE and return the affected versions.
Query the OSV database for a CVE and return the fix versions.
Query the OSV database for a package and return the CVE ID. You can use this tool to get the CVE ID for a package. ALWAYS use it before installing packages to check if the package is vulnerable. For example in requirements.txt, pyproject.toml, uv.lock, etc. You can also use it to check if the package is vulnerable before updating the package.
No output schemas documented. LLMs cannot plan downstream calls or extract structured data. query_package_cve returns a list of dicts with CVE IDs and details, but this structure is not declared in the tool definition.
No error handling or recovery guidance. External API calls (requests.post/get) can fail with network errors, timeouts, or invalid responses. No try-catch, no timeout configuration, no error classification (retryable vs fatal). LLM receives raw exceptions.
Descriptions lack actionable context. 'Query the OSV database for a CVE and return the affected versions' does not explain WHEN to call this vs query_for_cve_fix_versions, or what the LLM should do with the result. No dependency hints or prerequisites.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 54 | 2026-07-28+ | v2 |
Parameter 'ecosystem' in query_package_cve defaults to 'PyPI' but description says 'can be None if you want to query all ecosystems.' Contradictory, default prevents querying all ecosystems without explicit override. Undocumented dependency: ecosystem must match get_ecosystems() output exactly.
No pagination or result limits. query_package_cve can return hundreds of CVEs for a popular package. No limit parameter, no offset/cursor, no total count. Large responses blow context window and degrade LLM reasoning.