This MCP server has critical definition quality gaps across nearly all dimensions. Tool names are action-verb compliant (get_, create_, list_, etc.), which is good, but descriptions are severely underdeveloped, most fall into the 20-100 char range without adequate WHEN/WHY context. Input parameters have type definitions via FastMCP's auto-inference, but critically: (1) NO tool provides documented output schemas, (2) parameters lack descriptions in the source code (FastMCP decorators only capture type hints, not human-readable param docs), (3) error handling is completely absent, all tools blindly call the Shodan API with no try-catch, validation, or recovery guidance, (4) no distinction between similar tools (e.g., dns_lookup vs reverse_dns could be confusing without better context), (5) security concerns including unvalidated API key sourcing and no credential isolation pattern. The codebase provides zero structured guidance for LLMs on expected response schemas, making downstream tool chaining unpredictable. Per-tool analysis shows consistent pattern across all 11 tools: bare verb_noun naming, minimal docstrings (34-200 chars), inferred schemas without explicit parameter descriptions.
No documented output schemas. Tools return raw Shodan API responses without declaring structure, field types, or required/optional fields. LLMs cannot predict response structure for planning downstream calls.
Parameter descriptions missing. FastMCP auto-infers types from Python type hints, but the codebase provides ZERO human-readable descriptions for parameters (query, limit, ip, domain, name). Pattern baseline: 100% of A+ tools have param descriptions.
Add explicit parameter descriptions to all tools. For each parameter (query, limit, ip, domain, name), include: format expectation, valid range/enum, and dependency hints. Example: 'ip (string, required): IPv4 or IPv6 address to query. Format: xxx.xxx.xxx.xxx or xxxx:xxxx:... Use get_ip_info to look up an IP after dns_lookup.'
Document output schemas as JSON Schema or markdown tables. For each tool, specify: field name, type, description, and whether it's included in every response. Example for get_ip_info: { ip (string): IP address queried, ports (array of integers): open ports found, org (string): ISP/organization, country (string): geolocation }.
Wrap all API calls in try-catch blocks with categorized error handling. Return structured errors with actionable recovery: 'Invalid IP format. Expected xxx.xxx.xxx.xxx or IPv6. Did you mean: 192.168.1.1?' Avoid stack traces.
Fix API key sourcing. Require SHODAN_API_KEY environment variable at startup; raise a clear error ('SHODAN_API_KEY not set') if missing. Remove hardcoded fallback. Consider using a dotenv file or vault for local dev.
Add input validation. For IP parameters, validate IPv4/IPv6 regex before sending to Shodan. For domain parameters, check valid TLD and length. Return 'Invalid domain: must be 1-255 chars and end with valid TLD (.com, .org, etc.)' on failure.
Implement pagination for tools returning lists (shodan_search, get_cves, list_alerts). Add offset/limit parameters and return { results: [...], total: N, has_more: bool, next_offset: M }. Document in tool description: 'Results limited to 20 per call; use offset to paginate.'
Score history
Overall score trend
↑ 8 points across a rubric change (v1 → v2)
44/100
Scored
Grade
Overall
Spec posture
Rubric
2026-09-22
F
44
2026-07-28+
v2
2026-03-09
F
36
-
v1
list_alertsread onlyauthsource verified55/100
List all existing network alerts.
on_demand_scanwriteauthsource verified55/100
Launch an on-demand scan on a target IP (requires upgraded API access).
Zero error handling. All tools call the Shodan API without try-catch blocks. Invalid inputs (malformed IPs, nonexistent domains) will trigger unhandled exceptions with stack traces instead of actionable error messages.
Insufficient tool descriptions. Average description length ~60 chars; rubric baseline is 194 chars. Tool descriptions lack WHEN/WHY guidance. E.g., 'Advanced search with filters and facets' does not explain when to use shodan_search vs get_ip_info, what filters are supported, or what facets are returned.
Dangerous API key handling. SHODAN_API_KEY sourced from environment variable with a fallback to hardcoded literal 'your_shodan_api_key'. If env var is unset, the hardcoded string will be passed to Shodan API, leaking the default placeholder and potentially using wrong credentials.
No input validation. Parameters like 'ip' in get_ip_info and 'domain' in dns_lookup are not validated for format (IPv4/IPv6 regex, valid domain TLD) before being sent to Shodan. Invalid input produces cryptic API errors instead of LLM-actionable feedback.
No pagination support. Tools like shodan_search and get_cves accept a 'limit' parameter but do NOT return pagination cursors, total counts, or mechanism to fetch the next page. If Shodan returns 1000 matches and limit=5, the agent cannot iterate through results.
Verbose, unfiltered API responses. Tools return entire Shodan API response objects including metadata, pagination fields, and internal IDs not needed by the LLM.
Tool composition gap: dns_lookup and reverse_dns are functionally similar (both DNS ops) but named inconsistently with other 'get_*' patterns. LLM may struggle deciding between them without richer descriptions explaining the distinction.
No dry-run for destructive tools. on_demand_scan and create_network_alert are WRITE operations that modify state, but they lack a confirmation or dry-run mechanism.
on_demand_scancreate_network_alert
Filter and reshape API responses. Strip Shodan's internal _shodan, _id, and pagination fields. Return only: ip, ports, services, organization, country, cves (if applicable). This reduces token waste by ~40-60% vs raw API output.
Add confirmation/dry-run for write operations. on_demand_scan and create_network_alert should accept a 'dry_run=true' parameter that returns 'Would scan IP: 192.168.1.1' without actually executing. This prevents agent mistakes.
Enhance tool descriptions. Rewrite each to 150-200 chars following pattern: 'ACTION: [what]. WHEN: [when to use vs similar tools]. RETURNS: [key fields].' Example: 'ACTION: Search Shodan for hosts matching a query. WHEN: Use this to find all IPs with specific open ports or services; use get_ip_info for details on a single IP. RETURNS: array of IPs with ports and services. Limited to 20 results; use offset for pagination.'
Add rate limiting and timeout guards. Set a 10s timeout on Shodan API calls. If agent calls shodan_search 50 times in 1 minute, return 'Rate limit reached. Wait 60s before retrying.' Prevents runaway agent behavior.
Log all tool invocations to stderr (JSON format). Include: timestamp, tool_name, parameters (excluding API key), response_status (success/error), latency_ms. Enable audit trails for compliance.
Consider renaming dns_lookup → resolve_domain and reverse_dns → resolve_ip for clarity. Current names are descriptive but dns_lookup could mean either direction to an LLM unfamiliar with DNS terminology.
Document API prerequisites and quota limits in tool descriptions. E.g., 'on_demand_scan: Requires Shodan upgraded API tier. Standard tier limited to 1 scan/month; premium unlimited.' Prevents agent from repeatedly hitting quota walls.
Return structured error hints. When a domain is not found, return 'Domain not found. Available similar: example.com, example.org' (if Shodan provides suggestions). Guides LLM to self-correct.
Add permission checks if multi-user. Gate write operations (create_network_alert, on_demand_scan) behind user/agent permission verification. Log who authorized each destructive action.