Loading ToolBench
A demonstration system showcasing Kong AI Gateway capabilities with multiple specialized agents (Support, Success) orchestrated through a central orchestrator, featuring MCP tool integration, LLM-based decision making, and comprehensive observability.
8 tools with consistent verb-noun naming (get_, create_, search_, draft_). All have descriptions (avg 120 chars, within baseline 34-392). Input schemas present with typed parameters. However, output schemas are NOT documented anywhere in the source, critical gap for LLM planning. Parameters lack enums where appropriate (e.g., risk levels, severity). Error handling and recovery guidance absent. No tool annotations (readOnlyHint/destructiveHint). Composition is reasonable but missing pagination for list-like tools (search_runbook). Security: no visible secret injection or permission gates.
Create a followup task with account name, owner, due date, and action items.
Draft a customer reply message based on account name, CSM, issue summary, renewal risk, and technical summary.
Retrieve customer account information including customer ID, account name, segment, health score, renewal date, CSM, and ARR.
Get the current status of an incident including incident ID, status, severity, customer impact, and ETA.
Retrieve open support tickets for a customer including ticket ID, severity, summary, incident ID, and status.
Get renewal risk assessment for a customer including risk score, level, and drivers.
Output schemas not documented. LLMs cannot plan downstream calls or extract required fields (e.g., does get_customer_account return ARR as a number or string? Is renewal_date ISO 8601?). This forces agents to guess field types and names.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). LLMs cannot distinguish safe reads from destructive writes without explicit hints. draft_customer_reply and create_followup_task are WRITE operations but lack destructiveHint.
search_runbook lacks pagination (limit, offset, page_size). If runbook database grows, returning all results will blow context window. No documented result limit or next_cursor.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 57 | 2025-06-18+ | v2 |
Health check endpoint returning server status.
Search for runbooks by query string, returning matching runbook results with ID, title, and summary.
Parameters lack enums and constraints. get_incident_status 'severity' and get_open_tickets 'severity' accept free-form strings instead of enums (e.g., critical|high|medium|low). LLMs will hallucinate invalid values.
No error handling or recovery guidance. If get_customer_account fails with 'customer not found', the response should suggest 'Try search_customers() with a partial name.' Currently no such guidance visible.