Loading ToolBench
This server has 2 tools with significant quality gaps. The `list_folders` tool has a minimal schema (empty input object) and lacks parameter descriptions. The `search` tool has extensive documentation in its docstring but critical issues: (1) the input schema uses non-standard JSON Schema (`type: 'dict'` instead of `type: 'object'`), (2) parameters lack type definitions in the schema itself (only present in verbose docstring), (3) no output schema is documented, (4) no error handling guidance for common failures (invalid folder, malformed criteria, IMAP connection failures). The parameter descriptions exist but are embedded in docstring text rather than schema annotations, making them invisible to strict schema validators. The `search` function is async but there is no indication of how cancellation or timeout errors are handled. No pagination support for potentially large result sets. The tool naming follows verb_noun convention (acceptable), but schema quality is below production standards.
Get a list of IMAP folders.
Search for emails in the specified IMAP folder and fetch selected components. DESCRIPTION: This function performs an IMAP search operation using the provided query criteria and returns matching email messages with requested components (headers, body). The search criteria support logical operations and various email attributes. PARAMETERS: folder: str Name of the IMAP folder to search in (e.g., "INBOX", "Sent", "Drafts"). Must be a valid folder name on the connected IMAP server. criteria: dict JSON-formatted search criteria with the following structure: { # Logical operators (optional): "or": [ # ANY condition must match {<condition>}, {<condition>} # ... more conditions ], "not": [ # Negates multiple conditions {<condition1>}, {<condition2>} # ... multiple conditions are treated as NOT <condition1> AND NOT <condition2> ], # Search conditions (optional, multiple keys = AND): "from": "string", # Sender email/name contains this string "to": "string", # Recipient email/name contains this string "cc": "string", # CC recipient contains this string "subject": "string", # Subject line contains this string "since": "YYYY-MM-DD", # Format: "YYYY-MM-DD", date on/after "before": "YYYY-MM-DD", # Format: "YYYY-MM-DD", date before } If criteria is empty or missing, matches ALL messages in the folder. fields: dict, optional Specifies which components to include in the results: { "headers": bool, # Include email headers (From, To, Cc, Subject, Date) "body": bool, # Include email body text } If None or empty, an empty messages list is returned. RETURNS: dict A dictionary with the following structure: { "messages": [ # List of message data dictionaries { "id": str, # Message ID as "id@folder" "headers": { # Present if headers requested "from": str, "to": str, "cc": str, "subject": str, "date": str, "message-id": str }, "body": str # Present if body requested }, # ... more messages ] } If no messages match or fields is empty, "messages" will be an empty list. BEHAVIOR NOTES: - String searches are case-insensitive and match substrings - Multiple criteria at the same level are combined with AND logic - Logical operators can be nested for complex queries EXAMPLES: # Search for emails from alice OR bob since 2023-01-01 criteria = { "or": [ {"from": "alice"}, {"from": "bob"} ], "since": "2023-01-01" }
Input schemas use non-standard JSON Schema syntax. The `search` tool declares parameters with `type: 'dict'` instead of `type: 'object'`, and uses bare type strings instead of proper JSON Schema type definitions. Parameters lack `required` array and proper type nesting.
No output schema documented for either tool. The `list_folders` tool returns a bare list with no documented structure (is it a list of strings or objects?). The `search` tool's return value is described only in docstring text, not in a machine-readable schema. LLMs cannot reliably extract fields from undocumented responses.
`list_folders` tool has empty input schema (`{}`) with no parameters documented or described. While this is technically correct (the tool takes no inputs), the schema lacks any top-level description of what the tool returns.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 42 | <=2025-11-25 | v2 |
| 2026-03-09 | D | 50 | - | v1 |
No error handling or recovery guidance. The code will fail with unhandled exceptions if IMAP credentials are invalid, the server is unreachable, a folder doesn't exist, or criteria are malformed. No try-catch blocks visible. LLMs receive raw exceptions instead of actionable error messages.
IMAP credentials (IMAP_SERVER, IMAP_PORT, IMAP_USER, IMAP_PASSWORD, IMAP_USE_STARTTLS) are declared as module-level variables set to empty strings. No evidence of server-side secret injection via environment variables or vault. Credentials must be injected securely, not passed as tool parameters or hardcoded.
No pagination support for `search` results. If a folder contains thousands of matching emails, the tool returns all of them in a single response, risking context window explosion. The response structure does not include a total count or next_cursor.
Parameter descriptions in `search` tool are embedded in a lengthy docstring, not in the schema `description` field. Strict schema validators and IDE tools will not see these descriptions. Schema `description` fields must be populated for each parameter.
The `criteria` parameter accepts arbitrary dict structures with undocumented nesting rules (e.g., 'or' must be a list of dicts, 'not' must be a list of dicts). No schema validation of criteria structure. LLMs will pass malformed criteria that fail silently or crash the parser.
The `dsl_to_search()` function contains complex nested logic for converting criteria to IMAP search syntax, but has no input validation. Date format errors ('2023-13-45' instead of 'YYYY-MM-DD') will cause datetime.strptime() to raise an exception. No clear error classification or recovery guidance.