Loading ToolBench
The SciX MCP server has 22 tools covering astronomy literature search, paper retrieval, citation metrics, library management, and annotations. Tool naming is generally strong (all start with action verbs: search, get, create, delete, manage, export). Descriptions are present and substantive for all tools (ranging 50-150+ characters), explaining what each tool does and when to use it. However, schema quality is inconsistent: while all tools have input parameters with type declarations, parameter descriptions are often terse (single short phrases) and lack detail on constraints, formats, and valid ranges. Output schemas are not explicitly documented in the source code, responses are formatted as markdown or JSON but the structure of returned fields is not formally specified. Error handling is basic with no recovery guidance. Risk categorization (READ_ONLY, WRITE, DESTRUCTIVE) is helpful but not exposed in the formal tool definition. Overall, the server is above median due to consistent naming, present descriptions, and typed inputs, but lacks the depth of constraint documentation and output schema specification required for A-tier scores.
Add documents to a library based on a search query.
Create a new library with optional initial documents.
Delete an annotation (note) from a document in a library.
Delete a library permanently.
Edit library metadata (name, description, public status).
Export citations in 23 bibliographic formats (BibTeX, AASTeX, EndNote, IEEE, MNRAS, etc.) with support for custom formatting templates.
Output schemas not documented. Tool descriptions state response formats are 'markdown or json' but do not specify the structure of returned fields (field names, types, nesting). LLMs cannot plan downstream tool calls or extract fields without knowing response shape.
Parameter descriptions lack constraint details. For example, 'sort' parameter in search tool is described as "Sort order (e.g., 'citation_count desc')" but does not specify valid field names, whether ascending/descending keywords are required, or valid syntax. 'rows' and 'start' lack numeric bounds (minimum, maximum). LLMs cannot validate their own input without explicit constraints.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 58 | 2026-07-28+ | v2 |
Get an annotation (note) on a document in a library.
Get papers that cite a given paper (forward citations). Accepts a bibcode, DOI, arXiv ID, or SciX ID.
Get all libraries for the authenticated user. Can filter by type (all, owner, collaborator).
Get details about a specific library including metadata and list of documents.
Get citation metrics including h-index, citation counts, and paper statistics for a list of bibcodes.
Get detailed information about a specific paper by identifier: bibcode, DOI, arXiv ID, or SciX ID (scix:...).
Get the permissions for a library.
Get papers referenced by a given paper (backward citations). Accepts a bibcode, DOI, arXiv ID, or SciX ID.
Check the health and configuration of the SciX MCP server.
Perform set operations on libraries (union, intersection, difference, copy, empty).
Add or edit an annotation (note) on a document in a library.
Add or remove documents from a library.
Search SciX for astronomical literature. Supports full Solr query syntax including author:"Last, F.", title:keyword, abstract:keyword, year:2020-2023, and Boolean operators (AND, OR, NOT).
Search SciX documentation and help content.
Transfer ownership of a library to another user.
Update permissions for a library (grant/revoke access to collaborators).
Error handling provides no recovery guidance. Tools return errors but do not indicate whether the LLM should retry, ask the user, or call a different tool. For example, if 'get_paper' fails because the bibcode is invalid, there is no suggestion to call 'search' first. No error categorization (retryable vs fatal).
Destructive operations (delete_library, delete_annotation) lack dry-run or confirmation support. Agents should be prompted to confirm before executing irreversible deletions, but no such pattern is present. This risks accidental data loss.
Tool name 'manage_documents' and 'manage_annotation' are vague. The descriptions clarify actions (add or remove for manage_documents; add or edit for manage_annotation), but the names do not indicate directionality. Preferred: 'add_remove_documents' or separate 'add_documents' + 'remove_documents' tools. 'manage_annotation' conflates add/edit, consider 'create_annotation' and 'edit_annotation' as distinct tools.
No pagination limit enforcement documented. Tools like 'search' accept 'rows' parameter but do not state upper bound or default. Large result sets bloat context; tool description should cap results (e.g. 'rows: 1-100, default 20') and recommend pagination for larger datasets.
Response format parameter ('response_format': markdown or json) is repeated in 21 of 22 tools. This clutters tool schemas and forces LLMs to reason about format selection on every call. Consider: (a) removing the parameter and always returning structured JSON + helper method to format as markdown on the client side, or (b) documenting why both formats are necessary and which is preferred for agent use.
'bibcodes' parameter in tools like 'get_metrics', 'export', and 'manage_documents' accepts arrays, but descriptions do not specify maximum array size, whether duplicates are allowed, or what happens if an invalid bibcode is in the list. Constraints needed: 'List of 1-100 unique bibcodes; invalid codes are silently skipped' or similar.