Loading ToolBench
MCP server for document reranking using cross-encoder models to filter and rank retrieved documents by relevance
This MCP server has severe quality deficiencies across all rubric dimensions. The single tool 'execute_skill_action' lacks proper MCP integration, has an incomplete schema with no documented output structure, minimal descriptions, and the server implementation itself is non-functional (just prints JSON and exits). The tool definition exists only in documentation, not as actual registered MCP tool definitions in the code. The client.py is a mock implementation with hardcoded responses, not a real MCP server. No error handling, no input validation, no security checks, and no proper tool composition.
Execute cross-encoder reranking action on candidate documents
Tool definition not registered in MCP server code. Only exists in documentation. No actual tool registration, tool list, or MCP protocol implementation visible in mcp_server.py.
No output schema documented. Tool returns structured data (reranking_job_id, mean_reciprocal_rank_mrr, etc.) but this schema is not declared anywhere for the MCP interface. LLMs cannot plan downstream actions without knowing return fields.
Tool description is too minimal (58 chars). 'Execute cross-encoder reranking action on candidate documents' lacks context about WHEN to use this tool, WHAT it returns, or prerequisites. Baseline for A+ tools is 50-200 chars with clear selection criteria.
Parameter 'retrieved_docs_count' lacks range constraints. No minimum/maximum specified. LLMs could pass 0, -5, or 100000 with no validation. Baseline rubric requires 'page_size 1 - 100' style constraints in descriptions.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 25 | 2026-07-28+ | v2 |
Parameter 'top_k_filtered' lacks range constraints. No documentation of valid range (must be ≤ retrieved_docs_count? Must be ≥ 1?). Parameter description is generic.
No error handling or recovery guidance. mcp_server.py has no error response structure, no validation, no retry hints. If tool fails, LLM gets no actionable error message.
Server implementation is non-functional. mcp_server.py just prints a JSON message and exits. It does not implement the MCP protocol handshake, does not read from stdin, does not handle tool calls, and does not implement the stateless request/response loop required by MCP 2026-07-28 spec.
client.py is a mock, not an MCP server. It returns hardcoded synthetic data (reranking_job_id='hay_rrk_5519', mean_reciprocal_rank_mrr=0.965) without any actual reranking logic or external service integration. This is a test stub, not production code.
No tool composition or idempotency guarantees. No indication whether calling execute_skill_action twice with identical inputs produces the same result or different results. Agents need to know if a tool is idempotent to safely retry on transient failures.
No pagination or result limits documented. If reranking returns many results, no indication of how many are returned or how to paginate. Baseline rubric requires paginated_result pattern for list-like tools.