Loading ToolBench
VASP automation and analysis using CrewAI framework with MCP support
VASPilot has 6 tools with inconsistent quality. Tool names follow verb conventions (json_approx_search, analyze_crystal_structure, create_crystal_structure, wait_calculations), which is good. However, descriptions are present but vary in clarity and completeness. Input schemas are visible for most tools with type definitions and parameter descriptions present. The critical problem: parameter descriptions are often missing specific guidance on ranges, constraints, formats, and valid values. For example, 'struct_input' accepts 'string|Structure' but the schema shown only has type 'string', creating ambiguity. Error handling is not documented, no guidance on what happens if Materials Project API calls fail, if structure parsing fails, or how to retry. Output schemas are not documented in tool descriptions. The json_rag_tool.py excerpt cuts off mid-class definition, leaving JsonStrictSearch incomplete. Overall, tools have basic structure but lack production-grade rigor.
Analyzes crystal structure's space group and chemical formula. Takes a file path or pymatgen Structure object and returns space group information, chemical formula, lattice parameters, crystal system, point group, number of atoms, density, and elements.
Creates a crystal structure from atomic positions, element list, and lattice vectors. Optionally saves the structure to a VASP file.
Use RAG technology to search for relevant information from the JSON knowledge base.Can search for the most relevant configuration items and return short details.
Tool to query detailed descriptions. The detailed description is long, only query the most important tags.
Searches Materials Project database by criteria such as formula, elements, band gap, energy above hull, number of sites, space group, crystal system, and gap type. Optionally downloads structure files in VASP format.
Checks calculation task status and returns results. Polls the status of VASP calculations at intervals until completion.
Input schemas lack type specificity and constraints. 'struct_input' documented as 'string|Structure' but schema shows only 'string' type. 'search_criteria' is 'object' with no nested schema. Parameters lack minimum/maximum bounds, regex patterns, or enum constraints.
No output schemas documented. Tool descriptions do not explain what fields/structure the response contains. Downstream tools cannot reliably chain off outputs. Users must infer response structure from implementation.
No error handling guidance. Tools have no documented recovery paths. What happens if Materials Project API key is invalid? If structure file is malformed? If calculation ID doesn't exist? LLM gets no actionable error information.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 58 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 46 | - | v1 |
API credentials passed as parameters. 'search_materials_project' accepts 'api_key' as a tool parameter. Credentials in tool parameters appear in traces and logs, violating secret-injection pattern.
Parameter descriptions lack actionable constraints. 'top_k' has default=10 but no min/max bounds stated. 'positions' and 'lattice_vectors' accept arrays but no format documentation (3D coordinates? normalized? constraints on dimensions?). 'limit' on search has no upper bound guidance.
json_strict_search_tool description is vague ('Tool to query detailed descriptions...The detailed description is long...'). Does not explain WHEN to use it vs json_approx_search_tool. Ambiguous name distinction ('strict' vs 'approx' requires LLM to reason about similarity).
Source code provided cuts off mid-definition (json_rag_tool.py ends at 'class JsonStrictSea'). Cannot verify complete schema and implementation of json_strict_search_tool.