Loading ToolBench
MCP server providing access to Shodan API for internet-wide host scanning and reconnaissance
19 tools with consistent naming (verb_noun pattern: shodan_*) and basic descriptions (avg ~80 chars). All tools have input schemas with type definitions and parameter descriptions. However, descriptions are generic and lack LLM-optimized guidance on WHEN to use each tool vs alternatives. No output schemas documented. Error handling is minimal (only raise_for_status). API key exposed as optional parameter violates secret-injection pattern. No pagination guidance despite tools returning potentially large result sets. Missing tool annotations (readOnlyHint, idempotentHint). Per-tool scores average 52 due to these gaps.
Get information about the Shodan account linked to the API key.
Returns information about the API plan belonging to the given API key.
List all datasets that are available for download.
Get information about a specific dataset available for download.
Search within a dataset available for download.
Look up the IP address for the provided list of hostnames (comma-separated).
API key exposed as optional parameter violates secret-injection pattern. Credentials must use server-side injection via environment variables only, never as tool parameters. Agent traces log all parameters, secrets leak into logs and prompt history.
No output schemas documented. LLMs cannot plan downstream tool calls or extract required fields without knowing response structure. Tools returning lists (shodan_host_search, shodan_query, shodan_data) lack pagination guidance (limit, offset, total_count, next_cursor).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 64 | 2026-07-28+ | v2 |
Look up the hostnames that have been defined for the given list of IP addresses (comma-separated).
Search Shodan without results, only returns the total number of results and facet info.
Returns all services that have been found on the given host IP.
Search Shodan using the same query syntax as the website and return up to 100 results per page.
List all search facets that can be used when searching Shodan.
List all ports that Shodan is crawling on the Internet.
List all protocols that can be used when performing on-demand Internet scans via Shodan.
List the saved search queries in Shodan.
Search the directory of search queries that users have saved in Shodan.
List the most popular tags for the saved search queries in Shodan.
Shows the HTTP headers that your client sends when connecting to a webserver.
Get your current IP address as seen from the Internet.
Ping a host to see if it is reachable from Shodan's servers.
Descriptions lack LLM-optimized guidance. Generic descriptions like 'List all ports that Shodan is crawling' (shodan_ports) do not explain WHEN to call this vs alternatives or what the agent should do with the result. Descriptions should be 50-200 chars and answer: What? When? Why?
No error handling guidance. Tools call resp.raise_for_status() but provide no recovery hints to the LLM. When a query fails, the agent has no guidance on whether to retry, ask the user, or try an alternative tool. Error responses must categorize as retryable, user-fixable, or fatal.
No tool annotations (readOnlyHint, idempotentHint, destructiveHint). All tools are read-only but this is not declared. Annotations help agents understand safety and retry semantics. Add @mcp.tool(readOnlyHint=True) to all tools.