OpenShell-first end-to-end evaluation lab for configured AI agents, MCP servers, skills, and MCP Apps
Ghostlab is an evaluation framework, not an MCP server. The repository contains example MCP servers (tiny-mcp.js) but the actual Ghostlab package (rehearsal/) is a testing harness. The two visible tools (echo, host_info) have minimal descriptions (10-20 chars), no parameter descriptions despite having input schemas, and lack output schema documentation. Tool names are generic and lack action verbs. The codebase focuses on tool-call capture and failure classification, not on serving MCP tools itself.
Echo a message back to the caller.
Report the kernel the server is running on.
Tool descriptions are critically short (10-20 chars). 'Echo a message back to the caller' and 'Report the kernel the server is running on' lack context for LLM selection and do not explain when/why to use each tool.
Parameter 'message' in echo tool has no description. LLMs cannot infer what format or constraints apply. Schema shows type:string but no guidance on length, encoding, or purpose.
host_info has empty input schema (no properties). Output schema is not documented. LLMs cannot predict what fields will be returned or how to chain this tool with others.
Tool names lack action verbs. 'echo' and 'host_info' are generic nouns, not verb_noun patterns (e.g. get_host_info, send_echo). LLMs struggle to infer intent from these names.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 31 | 2025-06-18+ | v2 |
No error handling guidance. Tools do not document what happens on failure, whether errors are retryable, or what the LLM should do next.