Loading ToolBench
Two tools with identical input schemas but inconsistent naming conventions and documentation quality. Tool 1 ('search') has widget-specific metadata and better UI integration context; Tool 2 ('get_mermaid_image') lacks output schema documentation. Both tools accept single 'definition' string parameter with minimal description. No error handling guidance, no parameter validation constraints (e.g., max diagram size, format validation), and no recovery guidance documented. Schemas are properly structured (type, properties, required fields, additionalProperties=false) but lack depth in parameter descriptions and output documentation.
Generate a PNG image for a Mermaid diagram definition.
Render or view a Mermaid diagram using the widget output template.
Inconsistent tool naming: 'search' does not follow verb_noun pattern and is semantically misleading. It does not search anything, it renders/displays a diagram. Should be 'render_mermaid' or 'display_mermaid_diagram' to reflect actual behavior.
Parameter description is generic and lacks formatting constraints. The 'definition' parameter only states 'The Mermaid diagram definition' with no guidance on format, encoding, size limits, or required syntax. Should specify: 'Mermaid diagram definition in standard Mermaid syntax (https://mermaid.js.org). Max 10,000 characters.'
No output schema documented. Tool descriptions do not explain what fields or structure the response contains. For 'search': response includes 'text', 'structuredContent', and '_meta' but this is not documented for the LLM. For 'get_mermaid_image': base64 PNG data is returned but max file size and encoding format are undocumented.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 62 | 2025-06-18+ | v2 |
| 2026-03-09 | F | 37 | - | v1 |
No error handling guidance. Tool descriptions do not indicate what errors can occur (invalid Mermaid syntax, rendering timeout, browser crash) or how the LLM should recover. No guidance for 'Invalid diagram definition' vs 'Rendering timeout' vs 'Browser failed to load'.
Missing input validation constraints in schemas. Schemas do not declare maxLength, pattern, or examples. Parameter 'definition' has no upper bound, malicious or accidental 10MB Mermaid strings could exhaust memory during Playwright rendering.
Tool descriptions lack context about when to use each tool. 'search' is for viewing/rendering (UI widget); 'get_mermaid_image' is for image export. This distinction is not stated in descriptions, forcing LLM to infer from implementation details rather than documented intent.
Tool 'get_mermaid_image' description is 67 characters, acceptable length but lacks actionable detail. Does not mention: supported diagram types, output format guarantees, rendering timeout, or when to use this vs 'search' tool. Compare to baseline: A+ tools average 194 chars with full context.