Loading ToolBench
MCP server for AI agent integration with primer security scanning and vulnerability analysis
The Primer MCP server exposes 27 tools with reasonable parameter schemas and descriptions. Most tools have non-empty descriptions (18-150 chars typically) and input parameters with types and descriptions. However, there are significant gaps: (1) No documented output schemas for any tool, LLMs cannot plan downstream operations without knowing what fields to expect. (2) Parameter descriptions are present but often generic ('Package name', 'Config value') without constraints, ranges, or actionable error guidance. (3) Several tools operate on overlapping concerns (allow_add/allow_remove vs policy_list/policy_check) without clear delineation. (4) No error handling guidance, tools return errors but do not tell the agent how to recover. (5) Some parameters lack enum constraints where they should have them (e.g., 'value' in config_set). (6) The scan tool is complex with many optional parameters but lacks guidance on interdependencies (e.g., when 'format' and 'output' are valid). This is a solid C+/B- effort, useful tools with adequate schema coverage but missing the LLM-optimized polish needed for production agent integration.
Add a package to the allow-list
Print all entries in the allow-list
Remove a package from the allow-list
Remove all cached vulnerability results
Show vulnerability cache entry count, total size, and oldest/newest entry
Emit shell completion script
Read a single config value
Print all config values
Write a config value
No documented output schemas for any tool. LLMs cannot determine what fields are returned or plan follow-up operations. Scanning tool returns vulnerabilities but no schema shows the structure (CVE IDs, severity, remediation steps, etc.). Without output schemas, agents waste tokens calling tools speculatively and parsing responses without structure.
Parameter descriptions lack actionable constraints and ranges. E.g., 'value' in config_set accepts any string with no guidance on format, length, or allowed content. 'ecosystem' enum is defined but other parameters like 'format' in scan have enum values but descriptions do not reinforce the constraint.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | 2026-07-28+ | v2 |
Check PATH order, shim health, cache, and model state
Diff staged manifest changes and scan newly added packages
Write .git/hooks/pre-commit to block vulnerable package additions
Show system info (PATH order, shims, cache, model) - alias for doctor
Generate shims and update PATH
Start an MCP server over stdio for AI agent integration
Move .primer-ignore / .primer-policy.toml into .primer/ and update .gitignore
Download or import an AI model
List registered models with path and size
Remove one or more models
Set the active inference target (path or ollama model)
Open the most recent report in the browser or a specific report file
Validate policy file syntax without running a scan
Show active rules and expiry status from policy file
Generate a Software Bill of Materials (CycloneDX or SPDX)
Scan a package or manifest file for known vulnerabilities
Remove shims and PATH entries
Watch manifest files and auto-scan on change
No error handling guidance. Tools can fail (e.g., 'Package not found', 'Invalid ecosystem') but descriptions do not explain what the agent should do next: retry, call a discovery tool, or ask the user. Per pattern:recovery-guide, errors must tell the LLM how to recover.
'mcp' tool is vaguely named and described. 'Start an MCP server over stdio for AI agent integration' does not explain what this does when called as an MCP tool itself, is it recursive? Does it spawn a subprocess? Description is 45 chars, below the 50-char baseline for clarity. Naming convention would be 'start_mcp_server' for clarity.
Overlapping tool concerns without clear delineation. 'allow_add', 'allow_remove', 'allow_list' manage an allowlist; 'policy_list' and 'policy_check' manage policy files. Both operate on exceptions/overrides but the naming does not clarify the relationship. An agent may not know when to use allowlist vs policy.
'init', 'info', 'doctor', 'uninit', 'migrate' are system administration tools with minimal descriptions (40-60 chars). 'init' description is only 40 chars: 'Generate shims and update PATH', does not explain side effects, prerequisites, or when an agent should call it.
Parameter interdependencies not documented. 'scan' tool has 'format' and 'output' params that are 'only valid with file parameter', but descriptions state this as a constraint, not as guidance. An LLM may pass 'format' without 'file' and receive an error without understanding why. Dependencies should be explicit in descriptions: 'Use only when file is provided; otherwise the tool returns human-readable output to stdout.'