Loading ToolBench
MCP server for searching and fetching documentation from LangChain, LLamaIndex, and OpenAI docs
Single tool 'get_docs' has a reasonable description (194 chars, within baseline 34-392 range) and clear verb-noun naming. However, critical gaps undermine quality: (1) Input schema lacks enum constraints for 'library' parameter despite supporting only 3 known values (langchain, llama-index, openai), invites hallucinated values. (2) No output schema documented, LLMs cannot plan downstream operations or know what fields to expect. (3) Parameter descriptions embedded in docstring, not in schema annotations, LLMs may not parse them reliably. (4) No error handling guidance, tool raises ValueError for unsupported library but provides no recovery path. (5) No pagination or result limits documented despite fetching multiple URLs, could return massive unstructured text, exhausting context. Follows basic naming pattern but lacks production-grade schema rigor.
Search the latest docs for a given query and library. Supports langchain, openai, and llama-index. Args: query: The query to search for (e.g. "Chroma DB") library: The library to search in (e.g. "langchain") Returns: Text from the docs
Missing enum constraint on 'library' parameter. Tool accepts only 'langchain', 'llama-index', 'openai' but schema declares it as free-form string. LLMs will hallucinate invalid values like 'pytorch' or 'tensorflow'.
No output schema documented. Tool returns unstructured text from multiple URLs concatenated together. LLMs cannot determine what fields are present, cannot chain to downstream tools, and cannot validate results.
No result limits or pagination. Tool fetches and concatenates text from multiple URLs without cap. Large documentation pages could return 50KB+ of unstructured text, exhausting context window and degrading LLM reasoning.
Error handling lacks recovery guidance. Tool raises ValueError for unsupported library but does not tell LLM what to do next or list available options. LLM cannot self-correct.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 45 | 2026-07-28+ | v2 |
Parameter descriptions only in docstring, not in JSON Schema. FastMCP may not expose these to LLM context. Schema should include 'description' fields for 'query' and 'library' parameters.