Loading ToolBench
FastAPI-based MCP server for connecting softpack spack building commands to external services and LLMs
The Softpack MCP server exposes 22 tools with moderate definition quality. Most tools have verb-based names and brief descriptions, but significant gaps exist: (1) Parameter descriptions are often missing or trivial; (2) Output schemas are not documented in the source; (3) Error handling guidance is absent; (4) No tool annotations (readOnlyHint, destructiveHint, idempotentHint) despite clear risk categorization in the input spec. The server's domain is package management and recipe creation, which is well-scoped, but the implementations lack the precision expected of production tools. For example, 'install_package' and 'install_package_stream' are nearly identical tools differing only in transport mode, this violates the single-responsibility principle. Tools like 'uninstall_all_packages' and 'delete_session' are destructive but carry no confirmation mechanism or detailed error recovery guidance.
Copy an existing spack package to a session
Create a pull request for package modifications
Create a spack recipe from a PyPI package
Create a recipe (copy existing or generate new template)
Create a spack recipe from a URL
Create a new isolated session
Delete a session and clean up its files
Get git commit information for a package
Duplicate tool designs: 'install_package' and 'install_package_stream' both install packages with identical parameters; only difference is output streaming. This violates single-responsibility principle. Consolidate into one tool with a 'stream' boolean parameter or let the protocol handle streaming at transport layer.
Missing output schemas: No tool definitions include documented output/response schemas. LLMs cannot infer what fields are returned (e.g., does search_packages return 'id' or 'package_id'? how many results?). This blocks downstream tool chaining and forces LLMs to guess field names.
Destructive operations lack confirmation: 'delete_session' and 'uninstall_all_packages' are IRREVERSIBLE/DESTRUCTIVE but expose no confirmation step or dry-run capability. Agents cannot safely preview consequences before committing.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | D | 59 | <=2025-11-25 | v2 |
| 2026-03-09 | F | 0 | - | v1 |
Get checksum information for a package version
Get available versions for a package
Read a recipe file
Health check endpoint
Install a package using spack
Install a package using spack with streaming output
List all recipes in a session
List all active sessions
Request collaborator access to the repository
Search for packages in spack
Uninstall all packages in a session
Validate a package installation
Validate recipe content
Write a recipe file
Tool annotations missing: The input spec categorizes tools by risk (READ_ONLY, WRITE, IRREVERSIBLE, DESTRUCTIVE) but the MCP schema has no toolAnnotations (readOnlyHint, destructiveHint, idempotentHint). This metadata should be encoded in the schema so agents understand safety guarantees.
Minimal parameter descriptions: Many parameters have only 1-2 word descriptions (e.g., 'Session ID', 'Package name') that lack context for LLM selection. Descriptions like 'Session ID for isolated installation' (install_package) or 'PyPI package name' (create_pypi_recipe) are 3-5 words, well below the 50-200 character baseline for good descriptions.
No error recovery guidance: None of the tool descriptions explain what to do on failure (e.g., 'If install_package fails with version not found, try search_packages to find available versions'). Error responses are not documented.
Session_id parameter optionality inconsistent: Some tools mark session_id as optional (get_package_versions, get_package_checksum, get_git_commit_info), others require it (install_package, delete_session). The descriptions do not clarify when session context is needed vs. global scope. This ambiguity forces LLMs to reason about when to include the parameter.