Loading ToolBench
A FastAPI-based security analysis server that integrates VirusTotal API, Vectara RAG search, and Claude AI for comprehensive threat intelligence and malware analysis
This server implements FastAPI endpoints with Pydantic models and basic input validation, but falls short of production-grade MCP tool quality on several key dimensions. Tool descriptions are present but often generic or incomplete (e.g., 'research_domain' is truncated in source, lacking detail on what data is returned or when to use it). Input schemas are partially defined via Pydantic but lack proper enumerations for constrained values. Critical security issues: register/login/logout expose authentication concerns without explicit permission gating or error recovery guidance. Search and research tools lack pagination parameters despite potential for large result sets. Output schemas are not documented, LLMs cannot anticipate return structure or plan downstream calls. Error handling is minimal; no guidance on recovery strategies. Tool composition shows mixing of concerns (e.g., research_domain combines VirusTotal lookup AND Claude summarization in one tool). Named parameters are reasonable but descriptions are often vague (e.g., 'value' param for research_ip/research_hash/research_domain lacks format guidance, is it a partial match? Exact? Regex?).
Confirm user email registration via token from confirmation email link.
Download VirusTotal report as plain text file
Serve the home page HTML template.
Authenticate user with email and password, returning JWT access token.
Logout user by clearing client-side JWT token.
Register a new user with email, password, and optional full name. Sends confirmation email.
Research a domain using VirusTotal API and Claude AI summarization (truncated in source)
Tool descriptions are incomplete or truncated. research_domain, research_ip, research_hash are marked '(truncated in source)', full intent, return format, and composition (VirusTotal + Claude) cannot be assessed. LLMs will struggle to select the right tool or understand what data is returned.
Output schemas are not documented. None of the tools define what fields the LLM should expect in responses. Without documented return types, LLMs cannot plan downstream tool calls, extract correct fields, or handle errors. This violates the pattern baseline (100% of A+ tools have documented return types).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 49 | 2026-07-28+ | v2 |
Research a file hash using VirusTotal API and Claude AI summarization (truncated in source)
Research an IP address using VirusTotal API and Claude AI summarization (truncated in source)
Search the Vectara corpus for relevant documents using a query string.
Parameter 'value' used across multiple tools (research_domain, research_ip, research_hash, download_report_text) is ambiguous and lacks format guidance. E.g., research_ip says 'IP address' but doesn't clarify IPv4 vs IPv6, CIDR notation, or validation rules. Parameter naming should be more specific (domain_name, ip_address, file_hash) and descriptions must state format constraints (regex patterns, character restrictions, case sensitivity).
'home' tool is not an action verb. Naming should reflect the action (e.g., 'get_homepage', 'serve_home_page'). Current name 'home' is a route identifier, not a tool intent. Also, the schema exposes FastAPI Request object, framework internals should not leak into LLM-facing schemas; this should be abstracted away.
Research tools (research_domain, research_ip, research_hash) combine multiple concerns: VirusTotal API lookup + Claude AI summarization. This violates single-responsibility: agents cannot get raw data separately from summaries, and the tool description doesn't clarify which data is from VirusTotal vs. Claude. Should split into separate tools (e.g., lookup_domain_virustotal + summarize_domain_report) or document the composition clearly.
No pagination guidance. search_vectara has 'top_k' but no offset/cursor for fetching subsequent results. If searches return > 5 results, agents cannot retrieve them, design should include explicit pagination (offset/limit or cursor) and total count in response.
Authentication tools (register, login, logout, confirm_email) lack error recovery guidance. No documentation on what to do if registration email fails, token expires, or user is already registered. Description should guide LLM: 'If user already registered, ask user to reset password or login' rather than returning a raw 400 error.
Password parameter lacks format guidance. Description for 'password' in register tool should state: 'User password (minimum 8 characters, recommend mix of upper, lower, digit, special char)' or similar. Current description is bare 'User password', LLMs may pass weak values.
Login tool uses 'username' parameter but description says 'User email', naming mismatch. Should be 'email' to match semantic intent and avoid confusion with display names or user IDs.