The Relay server declares 40 tools with brief descriptions and basic input schemas, but exhibits significant quality gaps across naming, parameter documentation, and output specification. Tools are visible in the ToolsBrowser.js file with names, descriptions, and input parameters defined. However: (1) Many descriptions are under 50 chars and lack context for LLM selection (e.g., 'Read file contents as text' vs 'Read a text file from disk. Returns the full contents as a string. Use this to inspect code, configs, or logs.'). (2) Parameters lack validation constraints, no enums for constrained inputs (image_convert format, image_flip direction, image_rotate degrees, pdf_extract_pages page array), no ranges for numeric params, minimal format guidance. (3) Output schemas are not documented anywhere in the provided code, we cannot verify what fields are returned or what structure agents should expect. (4) Error handling is completely absent from descriptions. (5) Tool composition is poor: file_read, file_write, file_list, file_size, file_hash are five separate tools doing granular operations; a production agent would benefit from bundling (e.g., file_read + file_write into one file_manage tool or at least better naming). (6) Workflow tools (run_workflow, pm_plan, run_research, run_brand, run_ux, run_gtm, approval, assemble_plan) are vaguely described and their orchestration semantics are unclear, are these meant to be chained? Do they require human approval in between? The descriptions do not say. Per the rubric baseline of 194 chars average description length, most of these tools fall well short. Scoring reflects: naming is passable (verb_noun convention used consistently), descriptions are weak (mostly 10 - 40 chars), schemas are minimal (only input schemas visible, no output documented), and error handling is absent.
Human checkpoint between phases
Assemble final plan from phases
Convert CSV to JSON array
Format/pretty-print JSON
Query JSON with JMESPath
Convert JSON array to CSV
Compute SHA-256 hash of a file
List files in a directory
Output schemas are completely undocumented. The provided code shows only input parameter schemas; there is no documentation of what fields or structure each tool returns. LLMs cannot plan downstream tool calls or extract required data without knowing response shapes.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 53 | 2026-07-28+ | v2 |
Read file contents as text
Get file size in bytes
Extract files from a zip archive
Write text to a file
Create a zip archive from files
Convert image format (png, jpg, webp)
Crop an image to a region
Flip image horizontally or vertically
Convert image to grayscale
Get image dimensions, format, size
Resize an image to target dimensions
Rotate an image by degrees
Extract specific pages as new PDF
Extract text from PDF pages
Get PDF metadata and page count
Merge multiple PDFs into one
Count pages in a PDF
Split a PDF into individual pages
Phase 1: PM planning
Phase 3: Brand
Phase 5: Go-to-market
Phase 2: Research
Phase 4: UX
Execute a full 5-phase workflow
Decode base64 to text
Encode text to base64
Extract matches using regex
Convert markdown to HTML
Find and replace text
Count words, lines, characters
Fetch a URL and return content
Check HTTP status code of a URL
Descriptions are critically short and lack LLM-selection context. Most tool descriptions are 20 - 45 characters (e.g., 'Read file contents as text', 'Get image dimensions, format, size'). They do not explain WHEN to use the tool, what prerequisites exist, or what distinguishes it from similar tools. No error recovery hints.
Constrained-input parameters lack enums. Tools like image_convert (format: png, jpeg, gif), image_flip (direction: horizontal, vertical), image_rotate (degrees: 90, 180, 270), and pdf_extract_pages (page array) should declare enums or valid ranges in the schema. Currently they accept free-form strings, inviting LLM hallucination of invalid values.
Parameter descriptions are minimal. Most input parameter descriptions are 1 - 3 words (e.g., 'Path to file', 'Target width in pixels'). They lack format guidance, range constraints, or example patterns. Descriptions should state format, valid range, and when the param is required vs optional.
Workflow tools (run_workflow, pm_plan, run_research, run_brand, run_ux, run_gtm, approval, assemble_plan) have vague semantics. Descriptions like 'Phase 1: PM planning' and 'Phase 2: Research' do not explain: Are these chained? Does each require output from the prior? What does approval do exactly, does it block and wait for human input, or just log? Is approval idempotent? The orchestration model is completely opaque.
No error handling guidance. Tool descriptions and parameter documentation contain zero hints about error recovery: what to do if a file is not found, if a PDF is corrupted, if a regex pattern is invalid, or if the web endpoint is unreachable. Per pattern:recovery-guide, errors should tell the LLM what to do next.
No tool annotations. The feature list shows toolAnnotations=false. Production-grade tools declare readOnlyHint, destructiveHint, and idempotentHint in their schemas so LLMs know which operations are safe to retry and which are irreversible. file_write, file_zip, image_resize, pdf_merge, etc. should be marked destructive; file_read should be marked readonly.
Tool granularity is poor in file operations. file_read, file_write, file_list, file_size, file_hash are five separate tools performing sequential micro-operations. A production agent would benefit from consolidation (e.g., file_read_or_list bundling both into a single 'explore' capability) or at least much richer descriptions explaining when to use each vs. others.
Pagination and result limits are not documented. Tools like file_list, pdf_extract_pages, data_json_query that return potentially large results should declare result limits and pagination semantics (limit, offset, or cursor). Currently there is no guidance on whether an LLM should expect 10 results or 10,000.