Loading ToolBench
Retrieval-augmented MCP server for structured university programme information
sunway-rag-mcp is a STDIO-only RAG server with 2 tools (rag.search, rag.get). Both tools have explicit schemas and descriptions, but quality is uneven. Naming follows verb_noun convention adequately. Descriptions exist but are terse (13-30 chars). Schemas are present and typed, but parameter descriptions are minimal. Error handling is basic (ValueError, KeyError) with no recovery guidance. No tool annotations (readonly hints), no pagination support, no output schema documentation. The codebase is well-structured with intent routing and reranking logic, but the tool interface itself is sparse by production standards. Composition is sound, two single-purpose tools. The server lacks per-request _meta logLevel support, tool annotations, and structured error messages that would help LLMs plan recovery.
Fetch a stored chunk by id (text + metadata).
Semantic search over Sunway programmes with filters and reranking.
Tool descriptions are too terse (13 - 30 characters). 'Semantic search over Sunway programmes with filters and reranking' lacks actionable context about WHEN to use this tool vs alternatives, what preprocessing happens (intent classification, programme matching, reranking), and what the returned 'score' field means. LLMs cannot reliably determine tool selection with such sparse descriptions.
No output schema documented. rag.search returns {'results': [{'id', 'text', 'score', 'metadata'}]} and rag.get returns {'id', 'text', 'metadata'}, but these structures are inferred from code, not declared in the tool definition. LLMs cannot plan downstream tool calls without knowing response field names and types. Baseline 100% of A+ tools document return types.
No error recovery guidance. rag.get raises KeyError('Document not found: {doc_id}') with no hint about what to do next (e.g., 'Try rag.search() to find valid document IDs'). rag.search raises ValueError for empty query or out-of-range top_k without suggesting valid constraints. Baseline: error responses must tell the LLM what to do next.
Inferred effective spec: 2025-06-18+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 47 | 2025-06-18+ | v2 |
| 2026-03-09 | F | 36 | 2024-11-05+ | v1 |
Missing tool annotations. Neither tool declares readOnlyHint, destructiveHint, or idempotentHint. Both tools are read-only and idempotent, but this is not signaled to the MCP client. The Risk field in metadata shows 'READ_ONLY' but this is not expressed in a standard MCP annotation format.
No pagination support. rag.search returns up to top_k results (default/max 50), but there is no cursor, next_token, or total_count. If Chroma returns > 50 matches, the LLM cannot fetch subsequent results without re-querying with a modified filter. For a searchable corpus, pagination is essential for exploration.
Parameter descriptions are weak or missing context. 'top_k' has only 'Number of top results to return', no explanation of why results are reranked, how the score is computed, or when to increase/decrease top_k. 'query' lacks guidance on expected format (natural language, keywords, full questions). Parameter descriptions should be 50 - 150 chars with actionable context.