Loading ToolBench
Documentation-as-Code Middleware — Markdown docs to Document Graph for AI agents
UBP registers 6 tools with basic schemas and descriptions, but the implementation has significant gaps in alignment with production tool patterns. Tool names follow verb_noun convention (search, get, list), but descriptions are minimal (70-90 chars, below the 194-char production baseline). Input schemas are present and typed, but lack granular parameter validation, enums for constrained inputs, and field descriptions below the 72-char param annotation baseline. Output schemas are not documented, the codebase shows tool handlers but no explicit output type definitions visible in the tool registration layer. Error handling is absent; there is no recovery guidance or error categorization. The tools are READ_ONLY and compose well (search → get-page → get-context), but lack pagination defaults, response size limits, and chaining IDs in outputs. No tool accepts human-friendly identifiers (names, emails), all require opaque IDs (doc_id). Security is sound (no credential parameters, no destructive tools), but audit logging is not visible. Overall: functional READ-only tools that would benefit from deeper descriptions, output schema documentation, and error recovery patterns.
Full-text search across documents using FTS5 (SQLite full-text search)
Get contextual information about a document including related documents and graph neighbors
Retrieve the document link graph structure for visualization and analysis
Retrieve a complete document page by ID with metadata and content
List all available documents with metadata
Semantic vector search across document graph using embeddings and hybrid ranking
Tool descriptions are too short (70-90 characters vs. 194-char production baseline). Missing WHEN to use, prerequisites, and dependency hints. E.g., 'ubp-search' lacks guidance on semantic vs. full-text trade-offs or when to call ubp-fulltext-search instead.
No output schemas documented. Tool handlers exist, but the MCP tool registration does not declare return types. LLMs cannot predict response structure or plan downstream calls. Missing documented fields like total_count, next_cursor for pagination, or per-item success/failure for batch operations.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 47 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 0 | - | v1 |
Parameter descriptions are absent or minimal. 'query' in ubp-search lacks guidance on valid syntax (boolean operators?, regex?). 'limit' lacks min/max constraints (LLMs may pass absurd values). 'max_depth' in ubp-get-context has no guidance on performance impact or recommended range.
Tools require opaque doc_id parameters with no human-friendly alternative (doc_name, doc_title). Users don't have doc_ids, they have document names or titles. This forces a multi-step lookup (list_pages → search → get_page) for simple queries. Violates the chat-data-model principle.
No explicit error handling or recovery guidance. Responses do not categorize errors as retryable, user-fixable, or fatal. Missing actionable error messages (e.g., if doc_id is invalid, what alternatives are available?).
ubp-list-pages returns results without documented pagination limits or defaults. At scale, returning thousands of documents wastes tokens and risks context exhaustion. Missing total_count or next_cursor for stateful pagination.
No constrained inputs (enums) for parameters that accept known sets of values. If ubp-list-pages supported sort_by or sort_order, these should be enums, not free-form strings. Helps LLMs pick valid options without hallucination.