Loading ToolBench
Intelligent tutoring system for KCA University that provides question retrieval, answer generation using local LLMs, and RAG-based past paper context augmentation
Server has 15 tools with basic schemas and descriptions, but significant quality gaps. Most tool descriptions are present (194-char baseline met for some) but lack actionable context for LLM selection. Parameter descriptions exist but are often generic. No output schemas documented. No error handling guidance. Tool naming is reasonable (verb_noun pattern mostly followed) but some tools conflate multiple concerns (e.g., answer_question_orchestrated combines RAG decision + retrieval + formatting + answering). No tool annotations (readOnlyHint, destructiveHint, idempotentHint). Schemas show types but lack constraints (enums, ranges, patterns). No pagination support despite tools returning lists. Security: no evidence of secret injection or permission gates.
Generate an answer for a question in the specified mode. Validates the mode, formats the question with appropriate prompt template, sends to LLM with appropriate temperature, and returns the generated answer
Answer a question using RAG + MCP orchestration. Decides whether to use RAG, retrieves relevant past papers if needed, formats context, sends to MCP with context, and returns answer
Generate an answer with optional context (for RAG integration). Combines context with question if provided
Generate answers for multiple questions
Answer multiple questions with RAG orchestration
Check if Ollama server is running and accessible
No output schemas documented. Tools return data but LLMs cannot plan downstream calls or extract fields reliably. E.g., get_unit returns 'List[Dict]' with no field definitions.
Tool 'answer_question_orchestrated' and 'batch_answer_questions_orchestrated' conflate multiple responsibilities: RAG decision logic, retrieval, formatting, and answering. Should split into separate tools (decide_rag, retrieve_context, answer_with_context) for composability.
Parameter 'mode' appears in 6+ tools but lacks enum constraint. Description says 'exam, local, global, mixed' but no formal enum in schema. LLMs will hallucinate invalid modes.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 54 | <=2025-11-25 | v2 |
Generate a random CAT (Continuous Assessment Test) quiz from past papers. Retrieves all questions for the unit, randomly selects num_questions, and returns them as a quiz
Generate an answer for a question using MCP and LLM. Validates the mode, optionally includes context (for RAG), routes to MCP client, and returns the generated answer
Get list of all available units
Get list of all available years
Get statistics about the question database
Get all questions for a specific unit
Retrieve questions from a specific unit using semantic search
Retrieve similar questions from past papers using semantic search
Decide whether to use RAG for this query. Decision factors: query length, presence of exam keywords, presence of technical terms, query specificity
No error handling guidance. Tools return None or fail silently (e.g., 'Failed to generate answer' in logs). No recovery hints for LLM (e.g., 'Try check_connection() first' or 'Retry with different mode').
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). All tools are read-only but this is not declared. Agents cannot reason about safety or retry semantics.