Loading ToolBench
Turn mathematical LaTeX papers into theorem dependency graphs for AI agents.
PaperGraph MCP has 18 well-named tools with consistent verb_noun patterns (get_, workspace_, validate_). All tools have descriptions (avg ~80 chars, within baseline 34-392). Input schemas are present and typed for all tools. However, descriptions are terse and lack context about when to use each tool, prerequisites, or recovery guidance. Parameter descriptions are minimal (e.g., 'Paper identifier' for paper_id). Output schemas are not documented, LLMs cannot predict response structure. Error handling exists (ToolError wrapping) but lacks actionable recovery hints. No tool annotations (readOnlyHint/destructiveHint) despite clear risk classifications in metadata.
Return PaperGraph version and reproducible launch diagnostics.
Open or initialize a persistent multi-paper workspace.
Normalize arXiv ID and URL inputs and return the safe next action.
Validate a raw user arXiv request and return the safe next action.
Add or replace an arXiv LaTeX project in the active workspace.
Add or replace a local LaTeX project in the active workspace.
Add or replace a born-digital PDF paper in the active workspace.
Output schemas not documented. LLMs cannot predict response structure (e.g., what fields workspace_get_paper returns, whether results are paginated, what IDs are included for chaining). Forces agents to guess or make exploratory calls.
Descriptions are terse (avg ~60 chars) and lack context. 'Return metadata and counts for one stored paper' does not explain when to call this vs workspace_list_papers, what 'counts' means, or what paper_id format is expected. Descriptions should be 50-200 chars with WHAT, WHEN, and prerequisites.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear risk classifications in metadata. workspace_add_* tools are WRITE operations but lack destructiveHint. Agents cannot infer safety without annotations.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 68 | 2026-07-28+ | v2 |
Return incoming or outgoing citation evidence for a stored paper.
Return dependencies of a globally identified stored theorem.
Explain how workspace dependencies were extracted for one theorem.
Return external result mentions for one stored evidence result.
Return metadata and counts for one stored paper.
Return proof dependency evidence for one stored evidence result.
Return one stored evidence result with source spans.
Return proof evidence for one stored evidence result.
List all papers stored in the active workspace.
List stored evidence results across the active workspace.
Search theorem titles and bodies across the active workspace.
Parameter descriptions are minimal. 'Paper identifier' for paper_id does not explain format (arxiv:2401.12345? local-uuid? filename?). 'Search query' for query does not explain syntax, field scope, or example. Descriptions should state format, range, and constraints.
Error handling wraps exceptions in ToolError but provides no recovery guidance. 'No workspace is open' is clear, but most errors (WorkspaceError, PdfExtractionError, sqlite3.DatabaseError) are opaque to LLMs. Errors should suggest next steps (e.g., 'Try open_workspace(path) first').