Loading ToolBench
MCP server exposing CSCS FirecREST to an agent
Five tools with strong, agent-optimized descriptions (150 - 400 chars each). All tools have clear verb-noun naming (submit_job, get_job_status, list_files, download_file, get_job_log). Input schemas are present with type declarations and descriptions for all parameters. Descriptions include WHEN to use each tool, prerequisites, and recovery guidance, matching the 50 - 200 char LLM-optimized baseline. Error handling is explicit: _fail() returns structured {ok, error, http_status, detail} objects. Weaknesses: no output schemas documented (tool responses are inferred from code); no parameter enums or constraints (e.g., 'state' field values not enumerated); no pagination support despite list_files potentially returning large results; tool annotations (readOnlyHint, destructiveHint) absent.
Fetch a file from the cluster and return its contents. Use this to read a file the user asked about, to inspect a job's output when you know its path, or to show a configuration or input file. Prefer get_job_log when you want a job's stdout or stderr — it is simpler and handles the paths for you.
Fetch a job's stdout and stderr. Use this to read what a job printed, to see why it failed, or to check its output when you know the job id. This is simpler than download_file when you want a job's output — the paths are looked up for you.
Report the current state of a job you previously submitted. Use this to answer "is it done?", "did it finish?", "how long did it take?", or to wait for a job before reading its output. Call it with a job id from submit_job. The `state` field is Slurm's own value: typically `PENDING`, `RUNNING`, `COMPLETED`, `FAILED`, `CANCELLED` or `TIMEOUT`. Only treat `COMPLETED` as success — and note that `COMPLETED` with a non-zero `exit_code` means the script ran but the command inside it failed, so read the log before reporting success. This reads the scheduler's accounting record, which is authoritative. Do not infer job state from anything else: polling a job too early raises an error saying no record exists yet rather than returning a misread status. If the result carries a `warning`, the accounting record was not yet complete when it was read — re-query before telling the user anything about the job's name or partition.
List a directory on the cluster's filesystem. Use this to explore where things are: find a job's working directory, locate input files, check whether an output was written, or discover a user's home directory. Call it before submitting a job that needs a path you are unsure about, and after a job finishes to see what it produced. `path` must be absolute (for example `/home`, or `/home/<user>`). Listing a file rather than a directory is an error, not an empty result. Each entry has `name`, `type` (`-` file, `d` directory, `l` symlink), `size` in bytes, `permissions`, `user`, `group` and `last_modified`.
Output schemas not documented. Tool responses are inferred from code (e.g., submit_job returns {ok, jobid, system, stdout_path, stderr_path, next_step}), but no formal schema is published. LLMs cannot plan downstream calls or validate response structure without explicit output documentation.
No parameter constraints or enums. 'system' accepts free-form strings (should enumerate valid clusters); 'state' field values in get_job_status are documented in prose (PENDING, RUNNING, COMPLETED, FAILED, CANCELLED, TIMEOUT) but not as an enum constraint. LLMs may hallucinate invalid values.
No pagination support for list_files. Directory listings could return hundreds of entries, but no limit, offset, or cursor parameters are offered. Large results risk context window exhaustion.
Inferred effective spec: 2025-06-18+.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | B | 70 | 2025-06-18+ | v2 |
Submit a batch job script to the HPC cluster and get back its job id. Use this when the user asks to run, launch, submit or start something on the cluster — an analysis, a simulation, a script, a command. `script` is the complete batch script text, including its `#SBATCH` header. Always send a full script: a minimal one needs at least a partition, e.g. #!/bin/bash #SBATCH --job-name=my-job #SBATCH --output=my-job.out #SBATCH --partition=part01 echo hello `system` is the cluster to run on (`part01`/`part02` are the partitions of the demo system `cluster`). If you do not know the system name, call list_files on a path you do know, or check the FirecREST documentation — do not guess. `account` is the scheduler project account, only needed if the cluster requires one; it is safe to omit. `working_directory` only matters when this server is configured for FirecREST v2 (`FIRECREST_API_VERSION=v2`) — v1 ignores it. For v2 it is **required and must be an absolute path**: call list_files first if you don't already know one. Verified against the v2 demo: a relative value such as `.` is accepted and the job really does run, but v2's own status and metadata responses echo it back unresolved, which makes the job's output unreadable by get_job_log/download_file afterwards — so this tool refuses it upfront rather than submitting a job whose output nothing can read back. Returns the job id, plus the remote paths where the job's stdout and stderr will appear. Submission is asynchronous but this waits for the scheduler to accept the job, so a returned job id is a real, queued job.
Tool annotations absent. No readOnlyHint, destructiveHint, or idempotentHint metadata. Agents cannot infer which tools are safe to retry (submit_job is NOT idempotent; get_job_status IS read-only). This forces agents to reason about side effects without explicit guidance.
Error recovery guidance incomplete. _fail() returns structured errors, but does not suggest next steps (e.g., 'Invalid system. Call list_files to discover available clusters'). Agents receive error data but no actionable recovery path.