Loading ToolBench
MCP server for Zotero library research assistant with semantic search, graph exploration, and intelligent summarization of academic papers
Agent-Zot defines 9 tools with reasonable naming and schema structure, but suffers from critical gaps in parameter descriptions, output schema documentation, and error handling. Tool names follow verb_noun conventions (zot_search, zot_summarize, zot_explore_graph, zot_manage_*), which is good. However, descriptions are verbose but lack actionable constraints for LLMs. Most critically: (1) output schemas are NOT documented anywhere in the source, we cannot verify what fields LLMs should expect from responses; (2) parameter descriptions are present but generic, lacking format constraints, ranges, and enums; (3) no error recovery guidance visible; (4) write operations (zot_manage_*) lack confirmation/dry-run patterns; (5) dependency documentation between tools is missing. The codebase shows only tool definitions and brief descriptions in quick-mcp-tool-test.py and server.py, but actual implementation details (error handling, output structure) are not exposed. This means we're scoring based on visible definitions only, actual runtime quality may be higher or lower.
ChatGPT-compatible fetch tool (alias for zot_summarize) - Fetch and summarize papers from Zotero
ChatGPT-compatible search tool (alias for zot_search) - Finding papers in Zotero with semantic search
Check the status of the agent-zot daemon and underlying backend services (Zotero, Qdrant, Neo4j, Graphiti)
Smart graph exploration with automatic intent detection (citation/collaboration/concept/temporal/influence/venue), parameter extraction from natural language, and optimal Neo4j traversal selection across seven modes: Citation Chain, Influence (PageRank), Related Papers, Collaboration, Concept Network, Temporal, and Venue Analysis
Manage Zotero collections with operations to list, create, update, and delete collections
Output schemas completely undocumented. No visible response field definitions for any tool. LLMs cannot plan downstream tool calls or extract required data without knowing what fields to expect.
Parameter descriptions lack actionable constraints. 'query' described only as 'Search query in natural language', no guidance on length limits, valid characters, or format. 'limit' lacks min/max bounds (should specify 1-100 range typical for semantic search). LLMs cannot validate inputs locally and will pass invalid values.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 51 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 25 | - | v1 |
Manage the semantic search and knowledge graph databases with operations to show status, update database, rebuild database, and perform maintenance
Manage Zotero tags with operations to list, create, update, and delete tags
Smart intent-driven search with automatic intent detection (entity/relationship/metadata/semantic), backend selection (Fast/Entity-enriched/Graph-enriched/Metadata-enriched/Comprehensive), query expansion for vague queries, quality-based escalation, and result provenance tracking
Smart depth-aware summarization with automatic depth detection (quick/targeted/comprehensive/full), cost optimization to prevent unnecessary full-text extraction, multi-aspect orchestration (4 key questions for comprehensive mode), and four modes producing summaries from 500-800 tokens (Quick) to 10k-100k tokens (Full)
Write operations (zot_manage_collections, zot_manage_tags, zot_manage_database) have no confirmation or dry-run support. An agent mistakenly calling 'delete all collections' will execute immediately with no undo path. No recovery guidance.
No error handling guidance visible. Descriptions do not mention failure modes, recovery steps, or what the LLM should do if a tool fails (retry? escalate? try alternative?). This violates the recovery-guide pattern.
Duplicate tools (search/zot_search, fetch/zot_summarize). The ChatGPT-compatible aliases add confusion, LLMs must reason about which to use, wasting cycles. If maintaining backward compatibility, document in descriptions why both exist.
force_mode enums (fast, entity-enriched, graph-enriched, etc.) lack descriptions in parameter schema. LLMs cannot explain to users what each mode does or when to choose one. Tool description explains them conceptually, but parameter descriptions should summarize per option.
zot_manage_collections and zot_manage_tags expose a multi-action interface (action enum). Tools doing multiple things (create AND update AND delete) combine concerns. Consider splitting into separate tools: create_collection, update_collection, delete_collection, list_collections for clarity.
No pagination documented. zot_search and zot_explore_graph return 'limit' results but no offset/page, next_cursor, or total_count fields mentioned. Large result sets risk blowing context windows without pagination support.
Tool composition dependencies not documented. E.g., zot_explore_graph can take an optional paper_key, but where does an agent get this key? From zot_search results? Requires users to infer the workflow. Add discovery hints: 'Use zot_search() first to find paper keys, then pass one to this tool.'