Loading ToolBench
MCP-based board game rulebook search and summarisation system
Server has 5 tools with complete input schemas and descriptions, but quality is inconsistent. Tool names lack clear action verbs (e.g., 'is_rulebook_cached' uses 'is' rather than 'check'; 'query_rulebook_and_summarize' combines two actions). Descriptions are present but generic, most lack WHEN to use guidance, prerequisites, or dependency hints. Parameter descriptions are minimal (e.g., 'top_k' lacks range/constraint info). Output schemas are undocumented; tools return strings rather than structured objects. Error handling is basic (returns error strings, no recovery guidance). No tool annotations (readOnlyHint/destructiveHint). Composition is reasonable, tools form a logical chain (find → download → index → query), but lack idempotency guarantees and chaining metadata.
Downloads a rulebook PDF from the given URL and store it locally. This tool downloads the PDF file and saves it under data/pdf/{sluggified_game_name}/rulebook.pdf. Useful for building a local cache. Args: url: The direct URL to the rulebook PDF. Returns: The local filesystem path to the saved PDF, or an error message string if failed.
Finds the official rulebook URL for a given board game. This tool queries several websites using the board game name and attempts to locate a direct download link to the game's rulebook PDF. Args: game_name: The name of the board game to search for. Returns: The direct URL to the rulebook PDF if found, or an error message string.
Build and persist a FAISS vector index for a board game rulebook PDF. Steps performed: 1. Locate the PDF at `data/pdf/{slug}/rulebook.pdf`. 2. Extract all text from the PDF. 3. Split the text into overlapping chunks. 4. Compute embeddings for each chunk. 5. Build a FAISS index from those embeddings. 6. Save the index to `data/index/{slug}/{slug}.faiss` and a chunk-ID→text mapping to `data/index/{slug}/mapping.pkl`. Args: game_name: The human-readable name of the board game. Returns: The FAISS index path or an error description.
Check if a rulebook PDF for the given board game is already in the local cache. Args: game_name: The name of the board game to check. Returns: The local filesystem path to the cached PDF and FAISS index if present, or an error message if not found.
Tool names lack clear action verbs. 'is_rulebook_cached' should be 'check_rulebook_cache'; 'query_rulebook_and_summarize' combines two actions (query + summarize) and should be split into 'query_rulebook' and 'summarize_passages'.
Output schemas are undocumented. All tools return plain strings; no structured response format is defined. LLMs cannot reliably extract chaining IDs (e.g., index path from download_rulebook) or parse error vs. success.
Parameter descriptions lack constraints. 'top_k' has no min/max (should be 1 - 20); 'game_name' has no format guidance (slugification rules unclear to LLM); 'url' has no validation hints.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 65 | <=2025-11-25 | v2 |
Query the FAISS index for relevant rulebook passages. Steps performed: 1. Load the FAISS index and its mapping from `data/index/{slug}/`. 2. Embed the user's natural-language query. 3. Perform a top-K nearest-neighbor search. 4. Return the matching text chunks. Args: game_name: The human-readable name of the board game. query: A free-form question or search term. top_k: Maximum number of text chunks to return (default: 5). Returns: A string containign up to `top_k` text passages matching the query, or the error message.
No tool annotations. Tools lack readOnlyHint (is_rulebook_cached, query_rulebook_and_summarize, find_rulebook_url are read-only) and destructiveHint (index_rulebook, download_rulebook modify state). Agents cannot infer safety/retry semantics.
Error handling is minimal. Tools return raw exception strings (e.g., 'Error indexing...') with no recovery guidance. LLM cannot determine if error is retryable, user-fixable, or fatal.