Loading ToolBench
MCP server that sends mail over SMTP, gated behind an allowlist and a human confirmation
smtp-mcp has solid naming (verb_noun convention applied consistently), reasonable descriptions, and proper JSON Schema definitions. However, parameter descriptions are minimal (often single-word or very short), output schemas are not documented, and error handling guidance is absent. Tool definitions are explicitly visible and well-structured, but lack the depth expected for production-grade agent tooling. The server demonstrates good security awareness (allowlist, confirmation gates) but does not reflect this in tool descriptions. Average parameter description length ~20-30 chars; most params are adequately typed but lack actionable context.
Forward a mail message
List the configured recipients allowlist
Preview a mail message without sending it
Send a reply to a mail message
Send a mail message
Get the status and configuration of the SMTP server
Parameter descriptions are minimal (1-5 words: 'Recipients', 'CC recipients', 'Message body'). LLMs cannot infer constraints, formats, or validation rules from these generic labels.
Output schemas are not documented. Callers cannot predict response structure, field types, or chaining IDs. Tools return unspecified results; LLMs must guess what data is available for downstream operations.
Error handling and recovery guidance is absent. Tool descriptions do not explain when operations fail, what errors are retryable, or what the LLM should do next. No distinction between user-fixable vs fatal errors.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 50 | 2026-07-28+ | v2 |
Security constraints (allowlist, confirmation gates, SMTP_ALLOW_SEND flag) are enforced server-side but NOT mentioned in tool descriptions. LLMs cannot reason about why a call might fail or what preconditions must be met. Tool descriptions should state: 'Requires SMTP_ALLOW_SEND=true, recipient on configured allowlist, and human confirmation.'
Attachments parameter accepts an array but lacks description of file path format, size limits, or allowed types. 'File names to attach' is ambiguous, are these absolute paths, relative to a directory, or URLs? What is the max size per attachment?
Email recipient parameters (to, cc, bcc) are arrays but lack format specification. Are these email addresses, display names, or bare usernames? Should each entry be a string 'user@domain.com' or an object {email, name}? Current description 'Recipients' gives no guidance.
Tool descriptions do not explicitly state IRREVERSIBLE vs READ_ONLY nature. Per pattern:command-tool, destructive operations must declare consequences. 'Send a mail message' is passive; it should state 'This operation is irreversible, the message will be sent immediately and cannot be recalled.'
Pagination, result limits, and structure for list_allowed_recipients are undocumented. Does it return a flat array, objects with metadata, or a summary count? Is there a limit on how many recipients are listed?