Loading ToolBench
A learning planning agent that generates learning goals, creates learning plans, evaluates learning outcomes, and generates exam questions. It uses MCP servers for math operations and RAG-based document retrieval.
Three tools with basic definitions but significant quality gaps. All tools have input schemas and non-empty descriptions, but descriptions are minimal (10-80 chars) and lack actionable context. Tool names follow verb_noun convention but descriptions lack 'WHEN to use' guidance per pattern:tool-description. No output schemas documented. Error handling absent across all tools. Security concerns: RAG server hardcodes Azure credentials in source code instead of using server-side secret injection (pattern:secret-injection violation). Parameters have type definitions but lack detailed format/range documentation.
Add two numbers
Multiply two numbers
Use this tool when the user asks about python. Retrieve the most relevant documents from the python learning textbook.
Credentials hardcoded in source code (rag_server.py lines ~14-25). Azure API endpoint, deployment name, and API key loaded from environment but logged/visible in code. Violates pattern:secret-injection, credentials must never be visible in tool parameters or source control.
Tool descriptions are generic and lack 'WHEN to use' guidance. 'Add two numbers' does not tell the LLM when to select this over other arithmetic tools or what problem it solves. Per pattern:tool-description baseline of 194 chars, these descriptions (10-80 chars) are severely underdeveloped.
No output schemas documented. LLMs cannot plan downstream calls or extract results. Pattern:tool requires documenting return types, none are visible here.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 45 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 32 | - | v1 |
No error handling or recovery guidance. If retrieve fails (vector store not found, embedding service down, invalid query), the tool returns nothing actionable. Per pattern:recovery-guide, errors must tell the LLM what to do next.
'retrieve' tool name is vague and lacks action verb clarity. Naming patterns suggest 'search_python_docs' or 'query_learning_textbook', more specific than 'retrieve', which could mean many things. Per pattern:tool baseline, 90% of A+ tools start with clear action verbs (get, search, create).
Parameter descriptions lack format/range details. 'query' in retrieve is documented as 'The user's input query' but does not specify max length, format, or language expectations. Per pattern:constrained-input, format and constraints should be explicit.
No pagination or result limits declared. retrieve() returns up to 3 docs but no limit parameter or next_cursor mechanism. Per pattern:paginated-result, tools returning lists should support pagination and declare result limits in descriptions.
Math tools (add, multiply) have no use-case context. When would an LLM call these vs. calling a calculator or reasoning inline? Descriptions should explain domain constraints and when delegation is warranted.