The server provides 43 well-named tools with consistent verb-noun patterns (railway_list_*, railway_get_*, railway_create_*, etc.). Descriptions are present and moderately detailed (averaging 60-80 characters, which is below the 194-char production baseline). Input schemas are visible and mostly complete with Zod type definitions and descriptions. However, output schemas are not documented, responses are returned as raw JSON text without schema declarations. Error handling is minimal, with no recovery guidance or actionable error messages. No tool annotations (readOnlyHint/destructiveHint) are declared despite clear risk classifications in the brief. Missing pagination support on list operations that could return many results.
No output schemas documented. All tools return responses as unstructured JSON strings. LLMs cannot predict response structure, requiring them to parse text and guess field names, which increases hallucination and token waste.
No tool annotations declared. 43 tools have explicit Risk classifications (READ_ONLY, WRITE, DESTRUCTIVE) but these are not exposed via readOnlyHint, destructiveHint, or idempotentHint in the MCP tool definition.
Add output schema documentation for all 43 tools. Specify the structure of returned objects (e.g., project {id, name, createdAt, environments: [...]}), so LLMs understand what fields to extract for downstream calls.
Declare tool annotations for all tools: use readOnlyHint=true for READ_ONLY risk tools, destructiveHint=true for DESTRUCTIVE tools, idempotentHint=true for idempotent operations (idempotent hints enable safe retries for list/get operations).
Add pagination support to all list operations: accept limit (1 - 100, default 20) and offset/cursor parameters; return total count and next_cursor in response. This prevents context window exhaustion on large result sets.
Expand tool descriptions to 100 - 200 characters. Include: (1) what the tool does, (2) when to call it vs. similar tools, (3) key prerequisites or dependencies, (4) a note on what it returns. Example: 'Get a specific service's details including instance count, last deployment time, and connected GitHub repo. Call this after listing services to fetch full details. Returns service ID, name, status, and linked environment IDs.'
Add error recovery guidance. When a tool fails (e.g., 'Deployment not found'), return a message like 'Deployment XYZ not found. To find available deployments, call railway_list_deployments with projectId=ABC and serviceId=DEF.' This guides the LLM to self-correct.
Document parameter constraints in descriptions. E.g., for numReplicas in railway_update_service_instance: 'Number of replicas (1 - 100, typically 1 - 5 for cost efficiency).' For branch in railway_connect_service_repo: 'Git branch name (e.g., main, develop, feature/auth). Omit to use repository default.'
Score history
Overall score trend
↑ 9 points across a rubric change (v1 → v2)
58/100
Scored
Grade
Overall
Spec posture
Rubric
2026-09-22
D
58
2026-07-28+
v2
2026-03-09
F
49
-
v1
83/100
Create a new service in a Railway project. Source can be a GitHub repo, Docker image, or empty
Create or update multiple environment variables at once
List operations lack pagination. Tools like railway_list_projects, railway_list_services, railway_list_deployments have no offset/limit parameters or cursor support. Returning all results could exhaust context window or timeout.
Descriptions are too brief. Average description length ~70 chars vs. 194-char production baseline. Examples: 'Delete an environment permanently' (33 chars), 'List TCP proxies for a service in an environment' (48 chars). LLMs need more context about when to use tools and what they return.
No error recovery guidance. Tools return raw error messages without actionable next steps. E.g., a 'deployment not found' error should suggest 'List recent deployments with railway_list_deployments first.'
Missing parameter constraints. Many parameters accept free-form strings without validation. E.g., startCommand, buildCommand in railway_update_service_instance have no documented length limits, format restrictions, or examples of valid shells (bash, sh, node, python).
No dry-run or confirmation for destructive operations. Tools like railway_delete_project, railway_delete_service, railway_delete_volume execute immediately with no confirmation step. Agents can accidentally delete resources.
Add optional confirmation/dry-run for destructive tools. Offer a 'confirm' boolean parameter (default false) or a separate 'preview_delete' tool that shows what would be deleted before executing railway_delete_*. Alternatively, document in descriptions that these operations are irreversible and should be preceded by a manual confirmation step.
Include context-chaining IDs in responses. When creating a service or deployment, return not just the created object but also its parent projectId, environmentId, serviceId so downstream tools can be called immediately without extra lookups.
Add examples of common workflows in tool descriptions or docs. E.g., 'To redeploy after updating variables: (1) railway_upsert_variables(...), (2) railway_deploy_service(...). Both require serviceId and environmentId.'
Validate and document the Railway API token format. The client detects Project Tokens (UUIDs) vs. Account Tokens (prefixed). Document this in the server description and error messages if token auth fails.