Loading ToolBench
Deterministic MCP server for dependency and license inspection, policy compliance, and SBOM generation
This server demonstrates good foundation with all 6 tools properly registered via FastMCP, clear action verbs, and comprehensive input schemas. However, several definition quality gaps prevent a higher score: (1) Parameter descriptions are generic and lack actionable constraints (e.g., 'Path to the project root directory to scan' appears verbatim on 5 tools without usage guidance); (2) Output schemas are not formally documented in the code, the server returns Markdown summaries and JSON but doesn't specify the JSON structure for downstream LLM reasoning; (3) Error handling is minimal, tools reference DEMO_MODE fallback and OSV database but lack recovery guidance; (4) No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite all tools being read-only and safe to retry. Naming follows verb_noun convention well (analyze_, scan_, check_, generate_, suggest_), avoiding 'and' combinations. All 6 tools have descriptions 100+ characters and most parameters are described, which aligns with production baselines. The average tool scores 67/100.
Scan a project directory and return the full dependency graph (direct + transitive) for all detected ecosystems (npm, PyPI, Maven). Returns a Markdown summary plus JSON data.
Evaluate the project against a license and CVE policy. Use policy_path for a file path (e.g. demo_project/policy_strict.yaml) — plain text, no quotes. Or use policy: dict or JSON string path. Returns compliant (bool), violations list.
Generate a Software Bill of Materials (SBOM) in SPDX or CycloneDX format for the project.
List licenses for all project dependencies with risk flags (copyleft, unknown, multiple). Accepts a project path or an explicit list of packages [{name, version, ecosystem}].
Scan project dependencies for known vulnerabilities using the OSV database. Returns CVEs with severity and recommended fix versions. In DEMO_MODE or when offline, returns stub data.
Recommend alternative packages for problematic dependencies (e.g. GPL-licensed, with vulnerabilities, or unknown licenses).
Output schemas are not documented. Tools return Markdown summaries and JSON (per format_* helper calls), but the server code does not expose structured output type definitions for LLM reasoning. This violates the pattern:response-shaper requirement, LLMs cannot reliably parse unstructured text or infer JSON structure.
Parameter descriptions lack actionable constraints. 'exclude_dirs' appears on 5 tools with the identical description 'Directory names to exclude from scanning (e.g. node_modules, .venv, __pycache__). Default from config.', but does not specify: (1) whether the list is a suffix match or regex, (2) whether it is case-sensitive, (3) what the default from config is. LLMs cannot infer these details.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 62 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 17 | - | v1 |
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite all 6 tools being read-only and safe to retry. Agents cannot distinguish read-only tools from destructive ones without annotations, reducing confidence in multi-step planning.
Error handling lacks recovery guidance. scan_vulnerabilities states 'In DEMO_MODE or when offline, returns stub data' but does not explain how LLMs should interpret or validate stub data. No guidance on retryability, user-fixable vs. fatal errors, or fallback actions.
check_policy_compliance parameter 'policy' accepts both object and string (union type). Description states 'For API: use quoted string or object' but does not clarify: (1) what constitutes a valid quoted string (JSON? YAML?), (2) whether a string is a file path or inline content, (3) how to distinguish from policy_path. Ambiguity invites LLM misuse.
suggest_dependency_replacements has no parameters specifying what 'problematic' means (CVE severity threshold, license category, age, etc.). Description uses vague language: 'GPL-licensed, with vulnerabilities, or unknown licenses', but no tuneable thresholds for agents to control recommendations.