Loading ToolBench
A specialized Model Context Protocol server for Payload CMS 3.0, providing tools for validating Payload code, generating templates, scaffolding projects, and querying validation rules
This server has foundational tool registration but suffers from weak descriptions, incomplete schemas for complex parameters, and lack of output documentation. While 7 of 8 tools have explicit Zod schema registration in api/server.ts, most parameter descriptions are minimal (under 20 chars), and no tool documents its return schema. The 'scaffold_project' tool is marked as WRITE but has no confirmation/dry-run pattern. Generic enum values like 'templateType' lack context for when to use which variant. Error handling exists (try-catch blocks) but error messages are not actionable, they only return JSON stringification of exceptions. The server lacks guidance on tool composition and prerequisites (e.g., 'call query() first to discover validation rules'). Named after Payload CMS 3.0, the tools are domain-specific, but descriptions assume knowledge of Payload rather than teaching the LLM what each tool does.
Echo a message
Generate a complete Payload CMS 3 collection
Generate a Payload CMS 3 field
Generate Payload CMS 3 code templates
Execute SQL-like query against validation rules
Query validation rules for Payload CMS
Scaffold a complete Payload CMS 3 project structure
Validate Payload CMS code
Most tool descriptions are under 20 characters or vague domain-specific language. E.g., 'validate' = 'Validate Payload CMS code', 'query' = 'Query validation rules for Payload CMS', 'mcp_query' = 'Execute SQL-like query against validation rules'. LLMs cannot determine when to use validate vs query vs mcp_query without more context.
No output schemas documented for any tool. Tools return {content: [{type: 'text', text: ...}]} but the structure of the text content is undocumented. For 'generate_collection', does it return valid TypeScript code? For 'query', what is the structure of rules? LLMs cannot plan downstream operations or parse responses reliably.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 17 | 2026-07-28+ | v2 |
| 2026-03-09 | D | 54 | - | v1 |
Parameter descriptions are minimal. E.g., 'fileType' param in 'validate' tool is described as 'The type of Payload CMS file to validate', does not explain what validation rules apply or what errors might be returned. 'options' param in 'generate_template' is 'Configuration options for the template', too generic to guide LLM input.
'scaffold_project' is marked as WRITE (destructive) but has no dry-run, confirmation, or validation step before execution. An LLM could scaffold a full project with wrong database type, bundler, or structure and have no way to undo it. Missing pattern:confirmation-request.
Error handling returns raw exception messages in JSON without guidance on recovery. E.g., 'mcp_query' catches errors and returns {error: message} but does not distinguish retryable errors from user-fixable errors, and provides no recovery suggestions. A malformed SQL query error should suggest 'call query() first to discover schema'.
Tool composition and relationships undefined. 'query' and 'mcp_query' appear to overlap, when should the LLM use each? 'generate_template' vs 'generate_collection' vs 'generate_field', what's the relationship? No guidance on prerequisites or tool chaining.
'options' parameter in 'generate_template' uses z.record(z.any()), accepts arbitrary key-value pairs with no validation or schema. LLMs cannot discover valid options without external documentation, and invalid options fail silently.
Parameter 'fileType' in 'query' is marked optional but description does not explain behavior when omitted. Does it search all file types? Return an error? Default to 'collection'? Ambiguous optional params force LLMs to guess.