Loading ToolBench
MCP server for Open Science Framework (OSF) - search projects, registrations, preprints, and manage files
The server provides 11 tools with reasonably complete schemas and descriptions, but has several quality gaps that prevent a higher score. Most tools have clear descriptions (100-300 chars) and structured input schemas with type information. However, error handling is minimal, parameter descriptions could be more prescriptive (ranges, constraints, format hints), and output schemas are not explicitly documented. The tools are well-named with action verbs and follow sensible composition patterns (search, get, create, update, list, download, read, resolve). Authentication gating (write tools only if authenticated) is a positive security pattern. Main deficiencies: (1) No tool provides actionable error recovery guidance; (2) No explicit output schema documentation; (3) Some parameter descriptions lack format/constraint details (e.g., 'project_id' doesn't explain the expected format); (4) No per-parameter validation rules or bounds stated in descriptions; (5) No tool-level permission/scope declarations.
Create a new project on the Open Science Framework. REQUIRES AUTHENTICATION: You must have OSF_TOKEN set to use this tool. Creates a new OSF project with the specified attributes. Projects can be public (visible to everyone) or private (only visible to contributors). Categories: project (standard research project), component (sub-project/component), data (data repository), analysis (analysis project), software (code/software project).
Download a file from an OSF project and store it locally. Downloads the file and converts PDFs to Markdown for easy reading. Files are cached indefinitely for fast subsequent access. Supported conversions: PDF → Markdown (using PyMuPDF), Text files → stored as-is, Other formats → stored as binary.
Get detailed information about a specific OSF project. Returns comprehensive project metadata including title, description, contributors, tags, category, creation and modification dates, and links to files, wiki, and registrations.
Get the content of a wiki page from an OSF project. Returns the wiki page content in Markdown format.
List files and folders in an OSF project. OSF projects can have files stored in multiple storage providers: osfstorage (default OSF storage), github (connected GitHub repositories), googledrive (connected Google Drive), dropbox (connected Dropbox), figshare (connected Figshare), and others. Returns file metadata including file/folder name and path, file size and type, last modified date, and download URL.
No explicit output schemas documented for any tool. LLMs cannot know what fields to expect in responses, forcing them to parse unstructured output or make assumptions about return types. This violates the pattern:tool and pattern:response-shaper requirements.
Error handling completely missing from tool implementations. No error recovery guidance, no error categorization (retryable vs fatal), and no actionable error messages. Users of failed operations cannot recover. This violates pattern:recovery-guide and pattern:error-classification.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 60 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 0 | - | v1 |
Read the content of a previously downloaded OSF file. Returns the file content as text. For PDFs, returns the converted Markdown. Use download_file first to download and cache the file.
Resolve a DOI to find the corresponding OSF resource. OSF assigns DOIs to registrations (permanent, citable), preprints (for citation), and some projects (if minted). Given a DOI, this tool will extract the OSF resource ID, determine the resource type (project, registration, preprint), and return the resource details. Supported DOI formats: Full URL (https://doi.org/10.17605/OSF.IO/ABC12), DOI string (10.17605/OSF.IO/ABC12), OSF-style (OSF.IO/ABC12).
Search for OSF preprints. Searches across multiple preprint servers on OSF.
Search for OSF projects by query, tags, or category. Returns matching public projects (and private projects if authenticated).
Search for OSF registrations (pre-registered studies). Returns public registrations and user's own registrations if authenticated.
Update an existing OSF project. REQUIRES AUTHENTICATION: You must have OSF_TOKEN set to use this tool. Modify title, description, public status, and tags of an existing project.
Parameter descriptions lack prescriptive format/constraint guidance. For example, 'project_id' is described as 'The OSF project ID (e.g., 'abc12' from osf.io/abc12)' but does not explain length, allowed characters, or validation pattern. LLMs cannot infer these constraints from names alone. This violates review:param-validation-rules.
No declared permission/scope requirements on tools. Tools like create_project and update_project require 'REQUIRES AUTHENTICATION' note in description but lack formal scope declarations (e.g., 'read:projects', 'write:projects'). This prevents least-privilege agent configuration and auditing.
Numeric parameters lack bounds. For example, 'max_results' in search tools has no stated minimum/maximum in parameter descriptions (though code has MAX_RESULTS=50 globally). LLMs may pass invalid values. This violates review:param-validation-rules.
No pagination guidance in search tool descriptions. While 'max_results' parameter exists, descriptions do not explain how to retrieve additional results, whether pagination is supported via offset/cursor, or what the default limit is. This violates pattern:paginated-result.
No tool composition guidance. For example, 'download_file' requires a 'file_id', but 'list_files' returns files with metadata, the descriptions do not explicitly state that list_files results contain the file_id needed for download_file. This breaks pattern:tool-chain.
create_project description mentions 'REQUIRES AUTHENTICATION' but does not explain what error occurs if OSF_TOKEN is not set or what the LLM should do in that case. No recovery guidance.