Loading ToolBench
MCP server for scanning local paths and GitHub repositories for exposed API keys and secrets across multiple providers (OpenAI, Anthropic, Google Gemini, Azure)
This server has significant definition quality gaps across all dimensions. Tool naming is reasonable but descriptions are inconsistent. Input schemas are present but lack proper JSON Schema structure with types. No parameter descriptions visible in the schema definitions. Output schemas are not documented. Error handling exists but does not follow recovery guidance patterns. The server exposes potential security risks by validating actual API keys as parameters.
Recursively scans a local directory for secrets.
Scans a local path OR a GitHub URL via sampling.
Safely checks if a discovered key is active.
CRITICAL SECURITY: API keys exposed as tool parameters. The validate_key tool accepts 'api_key' as a direct parameter, which will be logged in agent traces, prompt history, and potentially echoed to users. This violates pattern:secret-injection.
No input schema types declared. Tool inputs show basic string type in descriptions, but the JSON Schema definitions lack 'type' field for parameters. Parameter 'provider' should be an enum (OpenAI|Anthropic|Azure OpenAI|Azure OpenAI Key|Google Gemini), not a free-form string. This invites hallucinated provider names.
No parameter descriptions in schema definitions. Each tool parameter lacks documentation of format, constraints, or examples.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | 2026-07-28+ | v2 |
No output schemas documented. Tools return JSON strings but do not declare the structure (fields, types, required). LLMs cannot plan downstream tool calls or extract correct data without knowing what to expect. Example: scan_directory returns 'total' and 'findings' but this is not formally documented in tool definition.
Sampling misuse. The code calls ctx.sample() to fetch GitHub content, but sampling (server-initiated model calls) is deprecated as of 2025-03-26. The server should instead fetch the GitHub URL directly via httpx or a GitHub API client, not delegate to the sampling API.
Inadequate error messages. Error responses are often bare JSON with 'error' field but do not guide the LLM on next steps. Example: 'Failed to fetch remote content.' does not tell the agent whether to retry, try a different URL, or ask the user. Per pattern:recovery-guide, errors should guide recovery.
Tool naming overlap and ambiguity. 'smart_scan' is vague (what makes it 'smart'?). The description mentions sampling but does not explicitly say 'Use this for GitHub URLs, scan_directory for local paths.' This forces the LLM to reason about tool selection instead of naming alone clarifying the distinction.
Missing dry-run or confirmation for sensitive operations. The validate_key tool calls live provider APIs with user-supplied credentials. No confirmation step or dry-run mode, an agent mistake could trigger rate limits, log credentials, or expose API activity.
No result pagination or limits documented. scan_directory recursively scans all files and returns all findings without pagination. If a large codebase contains hundreds of secrets, the response could exceed context window limits. No limit or pagination parameters offered.
Parameter relationships undocumented. validate_key's 'azure_endpoint' is optional, but the tool only validates it for Azure providers. The description does not state this dependency clearly, forcing the LLM to infer the relationship.