Loading ToolBench
MCP Server for Seer-AI Zotero Plugin - provides tools for research paper management, searching external scholarly databases, importing papers, managing collections, creating notes, and systematic review workflows
This server has 16 tools with comprehensive schemas defined via Zod, but several critical quality issues limit the score. Strengths: detailed parameter schemas with constraints (min/max, enums), generally descriptive tool descriptions, and proper typing. Weaknesses: (1) Three tools (terminal, code, process) are IRREVERSIBLE/WRITE but lack confirmation or dry-run patterns, creating catastrophic risk for an agent; (2) tool descriptions average 150 chars but some are vague ('Manage Zotero collections' lacks actionable when/why guidance); (3) error handling and recovery guidance are absent from tool specs; (4) no output schemas documented, LLMs must infer expected response structure; (5) several tools accept IDs-only (item_id, paper_id) without accepting human-friendly names, forcing extra lookup calls; (6) systematic_review tool is monolithic with 33 enum actions, violating single-responsibility principle; (7) no tool annotations (readOnlyHint, destructiveHint, idempotentHint) present despite WRITE/IRREVERSIBLE risks.
Check system environment for Python, Node, Git, and shell availability
Execute code in Python, JavaScript, or Bash
Manage Zotero collections (find, create, list, add/remove items)
Manage chat context items (add, remove, list)
Get metadata for a Zotero item
Import a scholarly paper into Zotero library
Create and edit Zotero notes
Three IRREVERSIBLE tools (terminal, code, process) lack confirmation/dry-run patterns. An agent can execute arbitrary shell commands or Python code without safeguards, enabling catastrophic damage (delete files, exfiltrate data, modify system). Production agents must have a confirmation_required pattern or explicit user approval gate.
No output schemas documented. Tool descriptions state WHAT they do but not WHAT they return. LLMs must infer response structure, leading to field-mapping errors. E.g., search_library returns results but the schema doesn't specify field names (title, authors, year, abstract, collection_id?), forcing LLMs to guess and fail when parsing results.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 64 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 0 | - | v1 |
Manage background processes (list, poll, wait, kill)
Read content of a Zotero item including notes and PDF text
Get citations and references for a paper using Semantic Scholar
Search external scholarly databases and repositories
Search query for titles, authors, abstracts, and full text
Comprehensive systematic review project management including protocol, extraction, analysis, and synthesis
Manage research tables (list, create, add papers, add columns, generate content)
Execute shell commands with optional background execution
Search and read web content
systematic_review tool violates single-responsibility principle. Its 'action' enum has 33 distinct operations (list_projects, create_project, add_papers, get_protocol, update_protocol, run_synthesis, etc.), making it a god object. This should be split into separate tools: list_reviews, create_review, add_review_papers, get_review_protocol, update_review_protocol, run_review_synthesis, etc. Agents cannot reason about which action to pick from 33 variants.
No tool annotations present (readOnlyHint, destructiveHint, idempotentHint). All 16 tools lack explicit risk metadata. This prevents agent frameworks from applying safety guardrails (e.g., warn before executing WRITE/IRREVERSIBLE tools). Terminal, code, and process especially need destructiveHint=true; search and read tools should have readOnlyHint=true.
Many tools accept only system IDs (item_id, paper_id), not human-friendly names. E.g., get_item_metadata requires a numeric Zotero item_id but users don't memorize item IDs, they know paper titles or authors. This forces an extra search_library call before every metadata lookup. Pattern: accept both user_id and user_name; resolve names to IDs inside the tool.
Error handling and recovery guidance absent from tool specs. No description of what errors are possible, what they mean, or how to recover. E.g., import_paper could fail if the paper ID format is invalid, the provider is down, or the item already exists, the tool spec says nothing about this. Agents cannot self-correct or plan fallbacks.
Descriptions for context, collection, table, web, and terminal tools are under 60 characters or generic. E.g., 'Manage chat context items' and 'Manage Zotero collections' lack actionable context. Short/generic descriptions prevent LLM tool selection.
import_paper accepts paper_id OR paper_ids but description doesn't clarify mutual exclusivity or whether passing both is an error. Similarly, search_external accepts provider OR providers, are both optional? If neither is set, does it search all? Parameter relationship documentation is missing.
terminal and code tools lack timeout or resource limits in descriptions. 'timeoutMs' and 'maxOutputBytes' parameters exist but no guidance on acceptable ranges. An agent could specify timeoutMs=0 (hangs forever) or maxOutputBytes=1GB (OOM). Describe realistic bounds in the parameter description.