Loading ToolBench
Monitors top PyPI and npm packages for new releases, diffs versions against previous releases, analyzes diffs with Cursor Agent for supply chain compromise indicators, and alerts Slack on malicious findings
This MCP server exhibits severe gaps in definition quality across nearly all evaluated dimensions. Of 24 tools, most lack proper descriptions (under 20 chars or generic), have incomplete or missing schemas, and show no evidence of input validation or error handling. The server appears to be a custom Python implementation without a visible MCP protocol wrapper, tools are inferred from function signatures in analyze_diff.py, package_diff.py, monitor.py, and other files, but no explicit MCP tool registration is visible in the provided source. This means the actual tool definitions that would be exposed to MCP clients are not shown, capping inference-based tools at a maximum of 50 per hard scoring rules. The toolkit is functionally designed to monitor supply chain threats but fails to meet production quality standards for LLM-agent integration.
Makes HTTP GET request to configured Slack webhook URL with optional query parameters
Uploads a file to Slack using the external upload URL flow (files.getUploadURLExternal -> POST -> files.completeUploadExternal)
Posts a message to a Slack channel with optional markdown text, thread reply, or block-formatted content
Writes diff report to temp workspace, runs Cursor Agent analysis, parses and returns verdict and analysis text
Queries PyPI changelog since a serial number, filters to watchlist packages with release events, prints releases with timestamps
Recursively collects all files under a root directory and returns mapping of relative-path to absolute-path
Tool definitions are inferred from Python source code, not explicitly visible as MCP tool registration. No MCP schema files, manifest, or server startup code shown. Cannot verify actual tool exposure to MCP clients.
Majority of tool descriptions are vague or under 50 characters. Examples: 'file_hash' (one sentence), 'collect_files' (one sentence), 'is_text_file' (no context on UTF-8 strictness). LLMs cannot decide when to select these tools.
Parameter descriptions are missing or minimal for many tools. Examples: 'unified_diff' context parameter lacks range guidance; 'send_slack_alert' slack boolean lacks explanation of toggle vs flag semantics; 'collect_files' output format (relative vs absolute path mapping) not explained.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 45 | <=2025-11-25 | v2 |
Downloads and diffs two versions of a package (both wheel and sdist when available), returns combined report and temp directory
Downloads a specific version of an npm package tarball from npm registry, handles scoped packages transparently
Downloads a specific version of a PyPI package (wheel or sdist) directly from PyPI JSON API without installing
Extracts .tar.gz, .tar.bz2, .zip, or .whl archives with path-traversal protection, returns root folder path
Fetches top N PyPI packages by 30-day download count from hugovk dataset, prints formatted table
Computes SHA256 hash of a file's contents
Generates markdown diff report comparing two versions showing added, deleted, changed files with unified diffs of changed text files
Queries PyPI JSON API to find the version released immediately before a given version
Determines if a file is text-readable (UTF-8 with strict error handling)
Reads saved PyPI changelog serial number from state file to resume monitoring from last checkpoint
Fetches top N packages from hugovk's PyPI dataset and returns mapping of package name (lowercase) to rank
Continuously polls PyPI changelog at specified interval, filters to watchlist packages, prints new releases in real-time until interrupted
Extracts verdict (malicious/benign) and reasoning from Cursor Agent output using regex pattern matching
Runs Cursor Agent CLI against a diff file with instructions to analyze for supply chain compromise, returns raw agent output
Performs single check of PyPI changelog from last ~10 minutes, prints matching watchlist package releases
Persists PyPI changelog serial number to state file for checkpoint resumption
Sends Slack notification for malicious package detection with details and links (only if slack=True)
Generates unified diff string for two text files with configurable context lines, returns None for binary files
No input validation constraints documented. Numeric parameters (context lines, top_n, lookback_seconds, interval) lack min/max bounds. Enum parameters (model, packagetype, ecosystem) not constrained as enums in schema or description.
Error handling guidance is absent. Tools like download_package, extract_archive, run_cursor_agent, and send_slack_alert have no documented recovery paths. LLMs cannot self-correct on failures.
Naming convention violations and ambiguity. 'SendMessage' and 'PostFile' use camelCase noun-first instead of snake_case verb_noun. 'GET' is a bare HTTP verb, not a domain-specific action. 'collect_files', 'load_watchlist', 'run_once' use weak generic verbs.
Output schemas are not documented. Tools like 'generate_report', 'collect_files', 'check_updates', 'fetch_top_packages', 'parse_verdict' do not specify what fields the LLM should expect in responses. LLMs must guess field names.
Unclear parameter semantics. Examples: 'send_slack_alert' slack boolean is confusing (toggle vs flag?). 'download_package' packagetype can be null with fallback logic not explained. 'monitor' and 'run_once' do not clarify if they block, spawn threads, or return immediately.
'GET' tool exposes low-level HTTP mechanics as a tool. This violates abstraction, HTTP GET to a webhook URL should be encapsulated inside 'SendMessage' or 'PostFile', not exposed as a separate tool for LLM invocation.
No pagination support documented. 'check_updates', 'fetch_top_packages', 'load_watchlist' may return large result sets but lack limit, page, offset, or next_cursor parameters. LLMs cannot page through results.