Loading ToolBench
City services Q&A MCP server with agentic RAG pipeline for answering questions about municipal services
CivicNav has 4 tools with basic schemas and descriptions, but significant gaps limit production readiness. All tools have descriptions (10-100 chars) and input schemas with type definitions, meeting minimum thresholds. However, descriptions are generic and lack LLM-optimized guidance on WHEN to use each tool vs. alternatives. Parameter descriptions are minimal (e.g., 'Search query' for civicnav_search.query). No output schemas documented, LLMs cannot predict response structure for downstream chaining. Error handling is absent; no guidance on retryability, user-fixable vs. fatal errors, or recovery paths. The civicnav_feedback tool accepts WRITE operations but lacks confirmation/dry-run patterns. Tool naming follows verb_noun convention (good), but lacks natural-identifier support, agents must work with opaque answer_ids rather than human-friendly references.
List all available service categories and the number of entries in each.
Submit feedback on an answer to help improve the system.
Ask a natural language question about city services. Returns an answer with source citations.
Search the city services knowledge base directly without synthesizing an answer.
No output schemas documented. LLMs cannot predict response structure (fields, types, pagination) for civicnav_query, civicnav_search, or civicnav_categories. This breaks tool chaining and forces agents to guess field names.
Descriptions lack LLM-optimized guidance. 'Ask a natural language question about city services' (civicnav_query) does not explain WHEN to use it vs. civicnav_search, what 'source citations' means, or what format the answer takes. Descriptions should be 50-200 chars with clear differentiation.
civicnav_feedback (WRITE operation) lacks confirmation/dry-run pattern and error handling. No guidance on retryability, user-fixable errors, or recovery. Agents cannot distinguish between transient failures and permanent rejections.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 47 | <=2025-11-25 | v2 |
Parameter descriptions are minimal or missing context. 'Search query' (civicnav_search.query) does not explain format, length limits, or example patterns. 'Number of results to return (1-20)' lacks guidance on when to use top_k=5 vs. top_k=20.
civicnav_feedback requires opaque 'answer_id' parameter. No guidance on how agents obtain answer_id from civicnav_query or civicnav_search responses. Broken tool chain, agents cannot naturally flow from query → feedback without extra discovery.