Loading ToolBench
Gmail MCP - Provides complete Gmail API access with file-based OAuth2 authentication
Gmail-MCP has a single tool (create_draft) with significant quality gaps. The tool has a description and detailed input schema, but critical issues undermine usability: (1) The schema description is confusing and contradictory ('raw' parameter is optional but overrides other params if provided), (2) Parameter descriptions lack clarity about mutual exclusivity and expected formats, (3) No output schema is documented, (4) The tool name 'create_draft' is appropriate but the implementation conflates RFC 2822 encoding with simple field parameters, creating a confusing dual-mode interface that will cause LLM misuse. (5) Error handling is generic (formatResponse with error field) and does not guide recovery. The code shows authentication error detection but returns confusing guidance ('re-authenticate by running: npx...') to an LLM that cannot execute shell commands. This is a single-tool server with a moderately complex but poorly-explained interface.
Create a draft email in Gmail. Note the mechanics of the raw parameter.
Confusing dual-mode parameter interface: 'raw' parameter claims to override 'to', 'cc', 'bcc', 'subject', 'body' if provided, but all parameters are marked optional. This ambiguity will cause LLMs to pass both raw AND simple params, resulting in either ignored params or failed calls. No explicit mutual-exclusivity constraint or enum-based mode selection.
No documented output schema. The tool creates a draft but the response structure is not defined in the tool definition. LLMs cannot predict what fields to expect (draft_id, thread_id, message, errors, etc.), forcing them to guess and handle parsing uncertainty.
Parameter descriptions lack expected format and constraint details. 'raw' is described as 'base64url encoded RFC 2822 format' but LLMs do not automatically understand RFC 2822 structure. No examples or patterns provided. 'to', 'cc', 'bcc' accept arrays but no length limits or email validation hints. 'includeBodyHtml' description is minimal and doesn't explain performance implications ('excessively large' is vague).
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 49 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 29 | - | v1 |
Error handling returns formatted responses with generic error messages. The authentication error response tells the LLM to 'run: npx @shinzolabs/gmail-mcp auth', which an LLM cannot execute. Error messages do not categorize failures as retryable vs. fatal, nor do they guide the agent toward recovery actions available within the MCP tool interface.
Tool description is generic and lacks guidance on when to use create_draft vs. alternative approaches (e.g. send directly vs. save as draft). No mention of prerequisites (valid OAuth2 credentials must be configured) or limitations (draft size limits, recipient constraints).
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) declared. The create_draft tool is not read-only but its idempotency is unclear, calling it twice with the same input will likely create two drafts, not idempotently return the same one.