Loading ToolBench
pbs-mcp has well-structured tool definitions with clear naming (verb_noun pattern: list_jobs, get_job, submit_job, delete_job, hold_job, release_job, list_queues). All 7 tools have descriptions and input schemas with Zod validation. However, descriptions are terse (10-60 chars), parameter descriptions lack actionable detail (e.g., 'Filter by PBS username' omits format/constraints), and output schemas are undocumented. Error handling returns raw SSH stderr without recovery guidance. No tool annotations (readOnlyHint/destructiveHint) despite clear risk levels (READ_ONLY, WRITE, DESTRUCTIVE). Parameters like 'extra_args' lack type clarity in descriptions.
Delete a PBS job via `qdel`.
Show full detail for a single PBS job via `qstat -f`.
Place a hold on a PBS job via `qhold`.
List PBS jobs via `qstat`.
List PBS queues via `qstat -q`.
Release a held PBS job via `qrls`.
Submit a PBS job by piping `script` to `qsub` over SSH. Returns the job id.
Output schemas undocumented. Tools return raw qstat/qsub output as unstructured text. LLMs cannot parse job IDs, statuses, or queue details without manual parsing. Structured output (job_id, status, queue, resource_list) would enable downstream tool chaining.
Error messages lack recovery guidance. SSH errors (e.g., 'qstat failed on headnode: command not found') do not tell the LLM what to do next. Should include: is this retryable? Should the user check the host? Is the PBS binary missing?
No tool annotations. delete_job and hold_job/release_job have clear side effects (DESTRUCTIVE, WRITE) but lack readOnlyHint/destructiveHint annotations. LLMs cannot distinguish safe read-only tools from dangerous ones without explicit hints.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 56 | 2026-07-28+ | v2 |
Parameter descriptions too terse and lack constraints. 'Filter by PBS username' (27 chars) does not explain format (email? short name?). 'extra_args' array lacks element type/format guidance. 'script' parameter (PBS batch script) lacks example structure or required directives.
No pagination or result limits documented. list_jobs and list_queues may return hundreds of items, bloating context. Should cap results (e.g., 50 jobs) and offer offset/limit parameters.