Loading ToolBench
FastAPI-based server providing AI-powered agent orchestration with integration to Google Drive, Notion, Gemini, and Zep for task management, memory, and multi-agent communication
This server exhibits significant quality gaps across naming, descriptions, schemas, and error handling. Of 25 tools, only 9 have visible implementations with proper schemas and descriptions (search_drive_pdfs, create_notion_task, query_notion_tasks, update_notion_task, gtd_capture, gtd_get_next_actions, gtd_promote_to_next, gtd_complete_task, chat, chat_with_audio, ingest_file, scrape_url, route_request). Of the 9 visible tools, naming is generally verb-correct but descriptions vary widely in quality (some lack parameter-level detail), and several tools lack documented output schemas. Tool descriptions range from adequate (search_drive_pdfs: 'Searches for PDF documents in Google Drive') to generic (chat: 'Text chat with a specific agent persona...' but missing return type documentation). Most critically, tools 12-23 (drive_list_files, drive_search, drive_upload_file, etc.) have zero visible schema definition, zero visible implementation, and descriptions that are placeholder stubs. Error handling across all tools is minimal, most lack recovery guidance, categorization, or actionable error messages per pattern:recovery-guide. No tool annotations (readOnlyHint, destructiveHint, idempotentHint) are present despite write operations (create_notion_task, update_notion_task, gtd_promote_to_next) being marked as WRITE risk.
Text chat with a specific agent persona (CEO, CTO, CIO, CFO, CMO, CAO, CRDO, or Context Engineer) with persistent memory via Zep.
Audio chat with agent: transcribe audio input and return JSON with transcription and response.
Classify a file for routing and processing (imported in main.py but implementation details truncated).
Create a task in the SmartDome Notion database. Returns the created page object or error dict.
Create a folder in Google Drive (imported in main.py but implementation details truncated).
Retrieve file content from Google Drive (imported in main.py but implementation details truncated).
13 of 25 tools (52%) have zero visible implementation, schema, or complete definition. Tools are imported in main.py but implementation details are truncated or missing (drive_list_files, drive_search, drive_upload_file, drive_get_file_content, drive_create_folder, send_agent_message, get_agent_messages, get_agent_routing_info, guard_output, inject_context, process_inbox_file, classify_file).
Output schemas are not documented for any tool. Tools like create_notion_task, query_notion_tasks, and chat do not specify what fields the LLM should expect in the response. Per pattern:tool and pattern:response-shaper, agents need to know the structure of returned data to plan downstream calls and extract relevant fields.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 42 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 32 | - | v1 |
List files from Google Drive (imported in main.py but implementation details truncated).
Search files in Google Drive (imported in main.py but implementation details truncated).
Upload a file to Google Drive (imported in main.py but implementation details truncated).
Retrieve messages for an agent from the agent bus (imported in main.py but implementation details truncated).
Get routing information for agents (imported in main.py but implementation details truncated).
Capture a raw task idea into the Notion inbox (GTD methodology).
Mark a task as complete in the Notion GTD system.
Retrieve the next actions list from Notion GTD system.
Promote a task from inbox to next actions (GTD workflow).
Output validation and guard filter (imported in main.py but implementation details truncated).
Ingest a file's text content into the agent's memory thread for context.
Context injection utility (imported in main.py but implementation details truncated).
Process and classify inbox files (imported in main.py but implementation details truncated).
Query tasks from the Notion database with optional filters. Returns: { success, count, tasks: [...] }
Analyzes query and returns the target agent_role string (lowercase) using Gemini routing logic.
Scrapes a URL and returns clean markdown-like text. Optimized for CMO analysis (Content, Headers, Links).
Searches for PDF documents in Google Drive.
Send a message to another agent via the agent bus (imported in main.py but implementation details truncated).
Update an existing Notion task by page ID. Only updates fields that are provided (non-None).
Error handling is absent or minimal across all tools. No tool provides recovery guidance, error categorization, or actionable messages. create_notion_task returns raw HTTP status and JSON from Notion API ('error': response.status_code, 'details': response.json()) rather than an LLM-friendly message like 'Invalid task status. Allowed: not_started|in_progress|completed'. Per pattern:recovery-guide, tools must tell the LLM what to do next.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) are present despite explicit WRITE risk classification. Tools marked WRITE (create_notion_task, update_notion_task, gtd_promote_to_next, gtd_complete_task, ingest_file, drive_upload_file, drive_create_folder, send_agent_message, process_inbox_file) should include destructiveHint=true so agents know they have irreversible consequences. Per Spec Alignment 2026-07-28, tool annotations are a current pattern and should be present for production readiness.
Parameter descriptions are sparse or missing. Tools like gtd_capture, gtd_promote_to_next, and gtd_complete_task have parameters (task_id, agent_id) with minimal or no description of what these fields control or what format they require. Per pattern:tool-description, every parameter needs a non-empty description explaining what it controls.
chat and chat_with_audio lack documented output schemas and return types. The source shows these tools call Gemini and return string responses, but the exact structure is not documented. LLMs need to know if the output is a plain string, JSON with fields like {role, content, metadata}, or something else. Without documented output schemas, agents cannot reliably extract or chain results.
Hardcoded credentials and secrets in configuration. app/tools/chat_tool.py loads GOOGLE_API_KEY and ZEP_API_KEY via environment variables (correct), but the code contains extensive hardcoded context and personas (GLOBAL_CONTEXT, AGENT_PERSONAS dict) which, while not credentials, reveals internal system architecture and is not ideal for a production tool. More critically, no tool validates or sanitizes user input before passing to downstream APIs. Per pattern:secret-injection and pattern:tool-gateway, credentials must not appear in tool parameters and all input must be treated as untrusted.
search_drive_pdfs returns mock data (hardcoded response with mock-drive-id-123, mock-drive-id-456) rather than real API results. The comment '# TODO: Implement actual Google Drive API call' indicates the tool is not functional. Agents will receive consistent fake results regardless of query, making the tool useless in production.
Tool composition is fragmented. Multiple tools operate on Notion tasks (create_notion_task, query_notion_tasks, update_notion_task, gtd_capture, gtd_promote_to_next, gtd_complete_task) without clear documentation of which tool to call for which workflow. Notion tools are in app/Execution/notion_tool.py and app/tools/notion_tool.py (two separate locations), suggesting duplication or inconsistent naming that could confuse agents.
No pagination support documented. query_notion_tasks accepts max_results (default 50) but no cursor, offset, or next_page field. If more than 50 results exist, agents have no way to fetch them. Per pattern:paginated-result, tools returning lists should accept page/offset and limit parameters and return a next_cursor or total count.