Loading ToolBench
Verified dependency audit as an MCP tool: a checked-against-the-installed-version verdict any AI agent can call, not a raw scan.
Three tools with clear, domain-specific names (audit_project, audit_dependencies, audit_repo) and substantive descriptions (150 - 250 chars each). All tools document their purpose and use case. However, input schemas lack formal type definitions and constraints, parameters are documented in descriptions but not in structured JSON Schema with explicit types, enums, or validation rules. Output schemas are not documented at all; the code references AuditPacket and grouped_product_findings but the MCP tool registration does not expose their structure to the LLM. Error handling is minimal, no recovery guidance or categorization. Naming follows verb_noun convention well; descriptions are above the 20-char floor but lack LLM-optimized structure (e.g., no 'When to use' or 'Prerequisites' sections). This is a solid domain tool hampered by incomplete schema formalization.
Audit a single lockfile and return a verified result. Convenience wrapper that delegates to audit_project.
Audit one OR MORE manifest/lockfiles together and return a verified result. This is the tool an agent working in a real checkout should reach for: it can pass both the manifest and the lockfile (the common local case), e.g. {"package.json": ..., "package-lock.json": ...}. Writing both into one tree is what lets closeread recover npm's direct-vs-transitive split -- a bare package-lock.json cannot mark top-level deps, so a declared production dep would otherwise collapse to transitive and the give-first lead would be lost. A single-file map (e.g. {"requirements.txt": ...}) works too.
Audit a remote GitHub repository by cloning it and running the full SCA scan. Returns a verified result with the same structure as audit_project.
Input schemas lack formal JSON Schema type definitions and constraints. Parameters are documented in descriptions but not registered with explicit types (string, object, array), enums, or validation rules (minLength, pattern, etc.). LLMs cannot parse constraints from prose descriptions alone.
Output schemas are not documented. The code references AuditPacket, grouped_product_findings, and _lead_dict but the MCP tool registration does not expose the return type structure. LLMs cannot plan downstream operations or extract fields without knowing what fields exist.
No error handling guidance. Tools do not document what errors can occur, whether they are retryable, or what the LLM should do next. A failed GitHub clone or malformed lockfile will return an exception with no recovery path.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 52 | 2025-06-18+ | v2 |
audit_dependencies description says 'Convenience wrapper that delegates to audit_project' but does not explain WHEN to use it instead of audit_project. LLMs need explicit guidance on tool selection criteria.
audit_project parameter 'files' accepts a mapping but does not document maximum file size, total payload size, or supported filename patterns. LLMs may pass oversized or malformed payloads.