Loading ToolBench
Gmail MCP has three tools with basic definitions and schemas. All tools have descriptions and input parameters with types, but descriptions are minimal (10-30 chars), parameters lack detail, output schemas are undocumented, and error handling is generic. The server returns JSON strings rather than structured responses, violating the pattern:response-shaper pattern. Tools are read-only (low risk), but parameter constraints are missing (e.g., max_results lacks min/max bounds). No tool annotations (readOnlyHint, idempotentHint) despite all being read-only operations. Missing dependency hints and recovery guidance in error messages.
Get the full content of a specific email.
List messages from Gmail inbox.
Search for emails based on different criteria.
Tool descriptions are extremely brief (10-30 chars). According to rubric baselines (p10=34, p90=392), descriptions under 20 chars cannot exceed 40 points. Descriptions lack context on WHEN to use each tool vs. similar ones. E.g., search_emails vs. list_messages distinction is unclear.
Output schemas are undocumented. Tools return JSON-encoded strings (e.g., 'json.dumps({"message": ..., "results": ...})') with no schema definition visible in tool registration. LLMs cannot infer response structure for downstream reasoning. Violates pattern:tool and pattern:response-shaper.
Parameter descriptions are generic or missing detail. 'Maximum number of results to return per page' (max_results in search_emails) lacks min/max bounds. No guidance on valid ranges. Rubric requires 'Specify minimum and maximum for numeric parameters (e.g. page_size 1 - 100, days 1 - 365)'.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 49 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 50 | - | v1 |
No tool annotations despite all tools being read-only. Rubric baseline: '100% of A+ tools have documented return types' and tool annotations (readOnlyHint, idempotentHint) are current patterns. All three tools should declare readOnlyHint:true.
Error messages are generic ('Error: Failed to search emails - {str(e)}') and provide no recovery guidance. Rubric requires: 'Error responses must tell the LLM what to do next.' E.g., when search_value is missing, error should suggest trying a different search strategy or consulting documentation.
No pagination documentation in search_emails despite implementing pagination logic (page, max_results, has_next, has_previous). Description should state: 'Supports pagination. Returns up to max_results emails per page.'
Tool output returns full email objects for get_email_content with no indication of structure. Risk of overwhelming context window if email is very long. No mention of truncation limits or when to use search_emails (metadata-only) vs. get_email_content (full).
search_emails requires both search_type and search_value but types (keyword, to, from) are not enumerated in the schema. Free-form string invites invalid values. Should declare as enum: ['keyword', 'to', 'from'].