Loading ToolBench
Three tools with explicit schemas and descriptions, but significant quality gaps. Tool naming is verb-based (run_test, new_test, get_command_output) which is good. However, descriptions contain example values (test.py, example.py, .hyperscale.json) that LLMs may reuse literally. Parameter descriptions are present but inconsistent in detail. The run_test tool has a 'quiet' parameter documented in the docstring but missing from the actual function signature, a critical mismatch. Output schemas are documented as JSON strings but lack field-level detail. Error handling is absent; no guidance on retryability or recovery. Security concern: the 'test' parameter accepts file paths without validation, risking path traversal attacks.
Get the output of an executed Hyperscale command. The run_id should be the run_id returned by the most recent Hyperscale-MCP new_test or run_test response and the task_name should be the name of the most recent Hyperscale-MCP tool call. Args: ctx: MCP context for providing progress updates task_name: Name of the task that we are checking. run_id: ID of the task run we are checking. Should be the run_id of the most recently run command. Returns: A JSON string of the current task run output. If the command is still running, the function will return a status of "RUNNING". Examples: "Show me the output of my latest hyperscale command" "What is the output of run 1234567890?" "What is the status of the current test run?" "Has the new test been created?"
Create a new Hyperscale test at the specified path. Args: path: The path at which to create the test (e.g. test.py, example.py). If any directories in the subpath do not exist, they should be created first. overwrite: If true, allow Hyperscale to overwrite any pre-existing file at the path. timeout: Optional timeout to create a new test. Default is None. If not specified, keep polling until result is returned and do not restart or cancel the test. Returns: A JSON string of containing the task_name and run_id to use in getting the command output.
Run the specified test at the given path. Args: test: Path to a *.py file containing the test (e.g. test.py, example.py) config: Optional path to a *.json file containing a JSON config (e.g. .hyperscale.json, config.json) log_level: A valid log level. Must be one of trace, debug, info, warn, error, critical, or fatal. Default is error. name: Optional name of the test (e.g. example, cicd, check_api). Default is default. quiet: If true, disables the stdout output workers: Optional maximum number of CPU cores to use for the test. Default is None. timeout: Optional timeout for the test. Default is None. If not specified, keep polling until result is returned. Returns: A JSON string of containing the task_name and run_id to use in getting the command output.
Parameter-docstring mismatch: run_test docstring documents 'quiet' parameter but function signature lacks it. LLMs will attempt to pass 'quiet' and fail.
Descriptions contain example file paths (test.py, example.py, .hyperscale.json) that LLMs tend to reuse literally. Use enum constraints or format patterns instead.
No input validation on 'test' and 'config' parameters. Paths are passed directly to subprocess without sanitization, risking path traversal or command injection attacks.
Output schema documented as 'JSON string' but no field-level detail provided. LLMs cannot infer structure of task_name or run_id fields or their types.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 53 | 2026-07-28+ | v2 |
No error handling or recovery guidance. Tools return JSON but no documentation of error cases, retryability, or what LLM should do if command fails.