Loading ToolBench
A conversational AI assistant with voice interaction capabilities, integrating multiple external services (weather, news, Wikipedia, web search, YouTube, WhatsApp) and a local RAG system for knowledge retrieval.
Vermeil's tool definitions are severely lacking in production-grade quality. While 10 tools are listed, most lack formal schema definitions, parameter descriptions, and output documentation. The codebase shows tool names and vague one-liner descriptions, but no explicit MCP protocol registration, JSON Schema input specifications, or structured output contracts. Tools are discovered via string matching in main.py ('if "weather" in lowered_input') rather than formal discovery. Parameter descriptions are minimal or absent (e.g., 'user_input' has a description but no type validation documented). Output schemas are not documented anywhere in the visible code. Error handling is completely absent, tools fail silently or crash. No tool distinguishes between retryable and fatal errors, provides recovery guidance, or validates inputs before execution. Security is a critical gap: tools like 'open_website', 'search_youtube', and 'open_whatsapp' execute system commands without sanitization, sandbox, or permission gates. The architecture is a string-matching router in main.py that calls functions directly, there is no formal MCP server implementation visible.
Retrieves user location information and returns it as context
Fetches current news and returns it as context
Returns system information including available RAM and CPU usage percentage
Returns the current date and time in a formatted string
Retrieves current weather information and returns it as context
Performs a web search for a given query and returns results as context
Searches Wikipedia for information about a given query and returns the result as context
Opens a specified website in the user's default web browser
Zero input schemas visible. No JSON Schema definitions for any tool parameters. Tools with parameters (get_wiki_context, get_web_search_context, open_website, search_youtube, open_whatsapp) declare 'user_input' but lack type validation, length constraints, format specifications, or enum constraints.
No output schemas documented. None of the 10 tools declare what fields they return, data types, or structure. LLMs cannot plan downstream calls or extract required fields. Violates baseline: '100% of A+ tools have documented return types'.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 30 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 37 | - | v1 |
Opens WhatsApp or initiates WhatsApp communication based on user input
Searches YouTube for videos matching a query and returns results or plays them
Vague tool descriptions (40 chars avg). Examples: 'Retrieves current weather information and returns it as context' (67 chars) lacks WHEN to use it, FORMAT of output, or DEPENDENCIES. LLM cannot determine if this tool returns current conditions only, forecast, alerts, or all three.
Destructive tools (open_website, search_youtube, open_whatsapp) execute without permission gates, dry-run options, or confirmation steps. No security model visible. 'open_whatsapp' passes user input directly to a system command, high command injection risk.
No error handling or recovery guidance. Tools fail silently or crash. No tool returns structured error responses with actionable next steps (e.g., 'User not found. Try search_users() first'). Violates pattern:recovery-guide and baseline: 'Error responses must tell the LLM what to do next'.
No MCP server implementation visible. Code shows main.py with a string-matching router (if 'weather' in lowered_input) calling Python functions directly. No MCP protocol handlers (CallTool, ListTools, etc.), no transport layer, no schema registration. This is a local monolith, not an MCP server.
Parameter descriptions missing or minimal. 'user_input' appears in 5 tools with identical vague desc 'The query to search for on X' / 'The website URL' / 'Contact information'. Violates baseline: '100% of A+ tool params have descriptions'. No format, length, or constraint guidance.
No input validation documented. Tools accept 'user_input' as unbounded free-form strings. No length limits, character restrictions, regex patterns, or sanitization guidance. 'open_whatsapp' and 'open_website' are especially at risk for command injection if inputs are passed to shell commands without escaping.
No tool composition or chaining support visible. If get_wiki_context returns a user_id or reference, there is no way to know what downstream tools accept it. No documented field mappings (user → user_id, etc.). Violates pattern:tool-chain baseline: 'Ensure tool A's output contains the IDs and references tool B needs'.
No idempotency guarantees. Repeated calls to 'open_website' or 'search_youtube' may have side effects (browser windows opened multiple times, duplicate searches). Agents retry on failure, tools must be idempotent or document side effects.