Loading ToolBench
Ground truth about npm and PyPI packages for AI coding agents. Detects hallucinated and slopsquatted dependencies before they get installed.
Three well-defined tools with strong descriptions (194 - 280 chars), comprehensive input schemas using Zod with proper enums and constraints, and documented output schemas. All tools are read-only with proper annotations. Naming is clear and verb-based (check_*). Minor gaps: parameter descriptions could be more explicit about format/constraints in some cases, and error handling guidance is minimal. Output schema documentation is present but could include more field-level descriptions.
Verify many npm or PyPI packages at once — use this before writing a package.json or requirements.txt, or handing a dependency list to a user. All names must belong to one ecosystem per call. Results are sorted worst-first so anything hallucinated or dangerous surfaces at the top. Read-only; batches registry requests and caches adoption figures, so 50 names take a few seconds.
Verify the packages a shell command would install or execute, before running it: `npm install …`, `npx …`, `pnpm add`, `yarn add`, `bun add`, `pip install`, `uv add`, `poetry add` and similar, including behind `sudo`, `&&` chains and `sh -c`. Extracts the package names (npm and PyPI at once), checks each against its registry, and returns them worst-first. A command that installs nothing by name (a bare `npm install` from a lockfile, `git`, `npm test`, or `npx <bin>` of a tool already in node_modules) returns total 0 and costs no network call — use check_dependencies on the manifest in that case. Read-only.
Verify a single npm or PyPI package before installing, importing, or recommending it. Returns whether it actually exists, and flags slopsquatting (a low-adoption package impersonating a popular one), names npm removed for malware, install-time scripts, deprecation, and abandonment. Call this whenever you are about to introduce a dependency you have not verified in this session. Read-only; one or two requests to the public registry. A verdict of UNKNOWN means the registry did not answer — retry, never assume safe.
Parameter 'cwd' in check_install_command lacks explicit format/validation guidance. Description says 'Defaults to the server's working directory' but does not specify if relative paths are allowed, required format, or validation rules.
Output schema fields like 'score', 'exists', 'complete', 'version', 'ageDays', 'weeklyDownloads', 'pointsTo', 'repository', 'didYouMean' are marked optional but lack clarity on when each is populated. LLMs cannot reliably plan downstream logic without knowing field presence guarantees.
Error handling is implicit (VERDICT enum includes UNKNOWN for unreachable registry) but tool descriptions do not provide explicit recovery guidance. E.g., 'If verdict is UNKNOWN, retry the call' is stated but no guidance on backoff, timeout, or fallback strategies.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 64 | 2025-06-18+ | v2 |
SIGNAL object includes 'id' field (e.g., 'npm_security_placeholder') but no enum constraint or documentation of valid signal IDs. LLMs cannot reliably parse or act on signals without a canonical list.
check_dependencies returns results 'sorted worst-first' but no explicit ordering guarantee documented in output schema. LLMs may not rely on sort order without explicit field or description.