Loading ToolBench
Slop and typosquat detection for PyPI packages - checks packages for supply chain risk before installation
Single tool 'assess_package' has a strong, action-oriented name and detailed description (280+ chars) that clearly states prerequisites and consequences. Input schema is present with proper type (string) and description. However, output schema is not documented, the tool returns a JSON string from assess() but the structure of CheckResult (fields: package, exists, pypi_unavailable, in_slop_db, risk_level, safe_to_install, recommendation) is not declared in the MCP tool definition. Parameter descriptions are adequate. No error handling guidance is visible in the tool definition itself, errors are returned as JSON strings without recovery hints. The tool follows pattern:tool and pattern:command-tool well but lacks pattern:response-shaper documentation.
REQUIRED: Check a package for supply chain risk before installing. You MUST call this tool before any `pip install`, `uv add`, or equivalent. If `safe_to_install` is false in the response, you MUST NOT install the package. Read the `recommendation` field and follow it exactly. Never skip this check or override a BLOCK result without explicit user confirmation.
Output schema not documented. Tool returns CheckResult as JSON string, but MCP tool definition does not declare the response structure (fields: package, exists, pypi_unavailable, in_slop_db, risk_level, safe_to_install, recommendation). LLMs cannot plan downstream actions without knowing what fields to expect.
No error handling guidance in tool description. When assess() returns an error (e.g., invalid package name, PyPI unavailable), the tool returns a JSON error string, but the MCP description does not explain error cases or recovery steps. LLMs cannot self-correct without explicit guidance.
Parameter 'package' accepts free-form strings but does not document format constraints. The description says 'e.g. requests or [email protected]' but does not state the allowed format, length limits (max 214 chars per code), or character restrictions (alphanumeric, dash, underscore, dot). LLMs may pass invalid values.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | <=2025-11-25 | v2 |
Tool description includes example values ('requests', '[email protected]') which LLMs may reuse literally in real calls.
No idempotency guarantee documented. The tool calls external services (PyPI, slop_db) which may have transient failures. If an LLM retries with the same package name, will it get the same result? Agents need to know if this tool is safe to retry.