Loading ToolBench
LLMDoc demonstrates solid definition quality with clear naming conventions, comprehensive parameter schemas, and well-structured tool definitions. All 5 tools follow verb_noun naming (search_, get_, list_, refresh_). Descriptions are present and substantive (100-200 chars). Parameter schemas include types, descriptions, defaults, and bounds. However, there are notable gaps: output schemas are not documented, error handling guidance is minimal, and parameter descriptions could be more prescriptive about format constraints. The tool set is coherent and well-composed for a documentation search domain.
Get document content with pagination support for large documents. For documents larger than 50KB, use offset/limit to paginate through content. The response includes has_more=True if more content is available. For targeted retrieval, use get_doc_excerpt instead.
Get relevant excerpts from a document based on a query. Returns targeted excerpts instead of full document content. Useful for large documents where you only need specific sections. Excerpts are ranked by relevance to your query.
List all configured documentation sources with statistics.
Manually trigger a refresh of all configured documentation sources. Fetches the latest documents from all sources and updates the index. Returns statistics about the refresh operation.
Search documentation and return relevant passages with source URLs. Use this tool when you need to find information about specific topics, APIs, or concepts. The search uses BM25 ranking for relevance.
Output schemas not documented. Tool descriptions explain inputs but do not specify what fields the response contains, forcing LLMs to infer structure from trial calls.
Parameter 'source' in search_docs accepts 'string|null' with no enum constraint. Free-form strings invite invalid source names. Should specify valid sources (e.g. via list_sources output or explicit enum).
No error recovery guidance. Tool descriptions do not explain what errors might occur (e.g. invalid URL, source offline) or how LLMs should respond. Callers will not know if a failure is retryable or fatal.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 69 | <=2025-11-25 | v2 |
list_sources has a generic description ('List all configured documentation sources with statistics') that lacks WHEN-to-use guidance. Should explain that this is a discovery tool used to validate source names before calling search_docs with a filter.
refresh_sources has a WRITE risk but no confirmation step or dry-run mode. If an LLM misunderstands its effect, it could trigger expensive re-indexing unintentionally. Should offer a preview or require explicit confirmation.