Precision-first analyzer for MCP servers and agent skills — measures the gap between what an artifact claims and what it does.
This is a benchmark/analysis tool, not a production MCP server. The 8 tools are sampled from a corpus of test fixtures (benign, malicious, and trap samples). Tool definitions are minimal: descriptions are present but extremely terse (10-50 chars, well below the 194-char baseline); parameter descriptions are entirely absent (null); no input validation, error handling, or output schemas are documented. The tools span calculator ops, HTTP requests, database queries, secret retrieval, and web fetching, a deliberately diverse test set, not a cohesive service. No tool exhibits the LLM-optimized descriptions, parameter constraints, or composition patterns expected of production tools.
Add two numbers.
Convert Celsius to Fahrenheit.
Get current weather for a city from a public weather API.
Fetch a web page over HTTP and return its text for the assistant to read.
Retrieve a stored secret by name from the local credential store.
Multiply two numbers.
Send an HTTP POST request with the given body to any URL you provide.
All parameter descriptions are null. LLMs cannot infer parameter meaning from names alone (e.g., 'a' and 'b' in add/multiply are ambiguous; 'name' in report/get_secret lacks context on format or constraints).
Tool descriptions are extremely terse (10 - 50 chars, vs. 194-char baseline). 'Add two numbers' and 'Multiply two numbers' lack context on when to use them, what they return, or error cases. Descriptions should be 50 - 200 chars and answer WHAT, WHEN, and WHAT IT RETURNS.
No output schemas documented. LLMs cannot plan downstream tool calls or extract required fields without knowing what each tool returns. E.g., does 'report' return a table, JSON, or text? Does 'get_page' return HTML, markdown, or plaintext?
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 31 | 2026-07-28+ | v2 |
Run a named read-only reporting query against the database.
No error handling or recovery guidance. Tools like 'post' (HTTP) and 'get_page' (web fetch) can fail in many ways (timeout, 404, SSL error, malformed URL) but provide no actionable error messages or retry hints.
Security concern: 'post' tool accepts arbitrary URLs and bodies with no validation, rate limiting, or audit trail. 'get_secret' exposes credential retrieval with no permission checks or logging. These are test fixtures, but production versions would violate secret-injection and permission-gate patterns.