Multi-LLM provider with MCP server support for tool calling across OpenAI, Anthropic, Bedrock, and Vertex AI
Critical failures across naming, descriptions, and schemas. Tool list contains 14 entries but only 6 unique tools (read_file, write_file, get_current_time, calculate, search_files, get_weather) with massive duplication (read_file appears 4x, calculate 4x, get_weather 3x). Descriptions are minimal (10-50 chars, well below 194-char baseline). Schemas present but sparse, most parameters lack type constraints, enums, or validation guidance. No error handling, no output schema documentation, no security considerations. Tool composition is poor: read_file and write_file lack path traversal protection hints; calculate lacks expression format constraints; get_weather lacks location validation. No pagination, no idempotency markers, no recovery guidance.
Perform a mathematical calculation
Perform a mathematical calculation
Perform a mathematical calculation
Perform a mathematical calculation
Get the current time in a specific timezone
Get current weather for a location
Get current weather for a location
Massive tool duplication: read_file listed 4 times, calculate 4 times, get_weather 3 times. Tool registry is corrupted or improperly deduplicated.
Descriptions are critically short (10-50 chars vs 194-char baseline). 'Read contents of a file' and 'Perform a mathematical calculation' lack context for LLM selection, prerequisites, or when to use vs alternatives.
No input validation constraints. 'calculate' accepts free-form 'expression' string with no format guidance (e.g., allowed operators, max length, syntax rules). 'get_weather' location param has no enum or validation. 'write_file' path param lacks path traversal protection hints.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 46 | 2026-07-28+ | v2 |
Get current weather for a location
Read contents of a file
Read contents of a file
Read contents of a file
Read contents of a file
Search for files by pattern in a directory
Write content to a file
No output schemas documented. LLMs cannot plan downstream calls or extract structured data. What does calculate() return, a number, a string, an object with result+unit?
No error handling or recovery guidance. If write_file fails due to permission denied or disk full, the LLM receives no actionable error message or retry strategy.