Loading ToolBench
A MCP server exposing croissant tools for converting Dataverse datasets to Croissant ML format
Server exhibits major quality gaps across the board. Of 8 tools examined, only 2 have complete input schemas with documented parameters. Most tools lack proper descriptions (under 20 chars or generic). Tool names are inconsistent in verb usage. No output schemas are documented. Error handling is absent. The codebase shows multiple competing server implementations (app/server.py, app/mcp-server.py, semantic_croissant/server.py, semantic_croissant/server_min.py) with no clear entry point. Tool definitions are scattered and partially inferred rather than explicitly registered in a central location.
Process a file in a dataset with DOI with datatool tool
Fetches a website and returns its content
Convert a dataset to Croissant ML format with get_croissant_record tool and explore the dataset with DOI or handle.
Say hello to a name
Get an overview of the Dataverse installations around the world sorted by country. Entrance point for the overview tools if no hosts are provided.
Get an overview of the Dataverse datasets statistics by host
Get an overview of the Dataverse files statistics by host
Multiple competing server implementations with no clear canonical entry point. app/server.py, app/mcp-server.py, semantic_croissant/server.py, and semantic_croissant/server_min.py all define tools differently. This fragmentation makes it impossible to determine which tools are actually deployed and creates maintenance confusion.
No output schemas documented for any tool. Tools return unstructured or partially documented responses. LLMs cannot plan downstream calls or extract fields reliably. For example, get_croissant_record likely returns JSON metadata, but the structure is nowhere documented.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | F | 46 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 45 | - | v1 |
Search for datasets in a Dataverse installation by host and optional query
Tool name 'datatool' is generic and unclear. Does it process, analyze, transform, or validate files? Verb-noun convention violated. Should be 'process_dataset_file' or 'analyze_file' or 'extract_file_metadata'.
get_croissant_record description (82 chars) is overly generic and confuses purpose: 'Convert a dataset to Croissant ML format with get_croissant_record tool and explore the dataset with DOI or handle.' The description repeats the tool name and mixes two distinct operations (convert + explore). Should be: 'Fetch a Dataverse dataset and convert its metadata to Croissant ML format, ready for machine learning tasks. Returns record ID and ML-ready schema.'
overview_datasets and overview_files have identical structure but no pagination parameters. If Dataverse installations have thousands of datasets or files, results will either be truncated silently or blow the context window. No limit, offset, or cursor provided.
search_datasets accepts optional 'query' parameter but no description of query syntax, supported fields, or example queries. LLMs cannot know whether to use Lucene syntax, SQL, natural language, or field:value format.
No error handling or recovery guidance. If a DOI is invalid, if a Dataverse host is unreachable, or if a file is missing, the tools will likely return bare HTTP errors or stack traces. No guidance on what to do next (retry, try alternative, ask user).
tool 'hello' (say hello to a name) appears to be a placeholder or demo and has no business logic relevance to the Dataverse/Croissant domain. Should be removed or this indicates incomplete refactoring.
datatool requires both 'doi' and 'file' parameters but the parameter descriptions do not explain: what file formats are supported? What does 'process' mean, parse metadata, extract data, validate, transform? Is 'file' a filename or a full path?
overview tool has no required parameters but description says it is 'Entrance point for the overview tools if no hosts are provided.' This is confusing, does calling overview return a list of hosts? Or is it just informational? LLMs will struggle to know when to call it.