Loading ToolBench
Open metadata for open-source packages across 100+ registries (npm, PyPI, crates.io, Go, NuGet, Maven, RubyGems, Packagist, Docker Hub, etc.) with cross-registry dependency and usage data. Keyless. A cross-registry complement to deps.dev and individual registry packs.
Three well-named tools with clear, action-verb prefixes (get_, lookup_, list_). Descriptions are concise and informative (80 - 180 chars), exceeding the 10-char minimum. All tools have proper JSON Schema with typed properties. However, critical gaps exist: lookup_package lacks required field validation (both purl and repository_url are optional with no mutual-exclusivity constraint documented), parameter descriptions are minimal (e.g., 'registry' lacks format/constraint guidance), and output schemas are completely undocumented, LLMs cannot predict response structure. Error handling returns generic error objects without recovery guidance. No tool annotations (readOnlyHint, idempotentHint) despite all being read-only, keyless operations.
Get a package's open metadata from ecosyste.ms — description, homepage, repository, license, latest version, monthly downloads, and dependent/usage counts. Works across 100+ registries. Keyless.
List the package registries ecosyste.ms supports (npm, PyPI, crates.io, Go, NuGet, Maven, RubyGems, Packagist, Docker Hub, and 100+ more) with their ecosystem and package counts. Keyless.
Look up every registry a package or repository is published to, via package URL (purl) or repository URL. Returns matching package records across all registries. Keyless.
Output schemas completely undocumented. LLMs cannot predict response structure (fields, types, nesting). get_package returns ~12 fields; lookup_package returns count + packages array; list_registries returns count + registries array. Without documented schemas, agents cannot plan downstream tool calls or extract data reliably.
lookup_package has no mutual-exclusivity constraint between purl and repository_url. Both are optional; schema does not enforce 'at least one required'. Description says 'provide purl or repository_url' but LLMs cannot parse English constraints, they need formal schema validation (oneOf, anyOf, or explicit required array).
Parameter descriptions lack actionable constraints. 'registry' parameter says 'default "npmjs.org"' but does not specify format, length, or valid values. 'name' parameter lacks guidance on scope syntax (e.g., '@vue/cli' vs 'vue/cli'). LLMs cannot infer these details from names alone.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 60 | 2026-07-28+ | v2 |
Error responses are generic and non-actionable. 'package not found' does not suggest next steps (e.g., 'Try list_registries to confirm the registry name' or 'Try lookup_package to search across all registries'). Errors lack recovery guidance, forcing LLMs to guess.
No tool annotations despite all three tools being read-only, keyless, and idempotent. Missing readOnlyHint and idempotentHint annotations prevent agents from optimizing retry logic and caching strategies.