Loading ToolBench
OSS license compliance auditor — AGPL contamination + NOTICE generation
Single tool 'scan' has a basic description (58 chars) but lacks critical detail on when/why to use it vs alternatives. Input schema visible but incomplete: 'target' accepts string|object without format constraints or examples; 'policy' enum values not formally declared in schema (only mentioned in description). No output schema documented. Parameter descriptions are minimal (target: 16 chars, policy: 24 chars), below the 72-char baseline. No error handling guidance, no recovery paths, no examples of valid inputs or expected output structure. Tool does not follow verb_noun pattern ('scan' alone is ambiguous, 'audit_dependencies' would be clearer). No indication of idempotency, side effects, or prerequisites.
Audit a manifest (path or parsed object) and return the report dict.
No output schema documented. LLMs cannot plan downstream calls or extract fields from the audit report dict. Rubric requires 100% of A+ tools to have documented return types.
Parameter descriptions are too short (target: 16 chars, policy: 24 chars). Baseline is 72 chars. 'Path to dependency manifest or already-parsed list' lacks format guidance (JSON file? YAML? Array structure?). 'Distribution policy preset' does not list valid options (proprietary, gpl-project, etc.) in the description.
Policy parameter lacks enum constraint in schema. Valid values (proprietary, distribute-binary, permissive-only, gpl-project, permissive-audit) are hardcoded in JavaScript but not exposed in MCP schema. LLMs cannot discover valid options without reading source code.
Tool name 'scan' is generic and ambiguous. Does it scan for vulnerabilities, licenses, dependencies, or malware? Verb_noun pattern (e.g., 'audit_dependencies', 'check_licenses') would clarify intent and help LLMs select the right tool when many are available.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 38 | 2026-07-28+ | v2 |
No error handling guidance. What happens if the manifest file is not found? If a license is unrecognized? If policy validation fails? LLM receives raw error with no recovery path (e.g., 'Try search_licenses() first' or 'Supported formats: package.json, requirements.txt, Cargo.toml').