Loading ToolBench
MCP server that finds automation which stopped working without telling anyone
Server has 7 well-named tools with clear, actionable descriptions (avg 156 chars, within baseline 194). All tools are READ_ONLY and properly scoped. Input schemas are present with type definitions and descriptions for all parameters. However, output schemas are not documented, responses are described informally but lack formal JSON Schema definitions. Error handling is minimal; tools do not guide recovery or categorize failures. No tool annotations (readOnlyHint, etc.) despite all being read-only. Parameter descriptions are good but some lack format/range constraints (e.g., max_age_hours, within_hours lack min/max bounds). The audit() tool is well-composed and demonstrates good intent, but individual tools could better support chaining by documenting expected downstream usage.
Run every check and return one prioritised report. This is the tool to call first. It answers: what on this machine is broken right now and nobody has been told?
Check an integration by its payload, not its status code. Set `require_json_path` to the part of the response that proves the integration is alive — for example "data" for a list of connected accounts. An endpoint returning HTTP 200 with an empty list is a disconnected integration, and this is the check that catches it.
Report logs that have not been written to recently. A daily job whose log is 26 hours old has not run. This is the cheapest liveness check available and it requires no instrumentation.
Scan the tail of recently-written logs for error signatures. Only recent files and only their last lines, so that failures fixed last week do not keep reappearing in the report.
Return only the scheduled jobs whose last run exited with an error.
Output schemas not formally documented. Tools return list[dict[str, Any]] or dict[str, Any] with no JSON Schema definition of response structure. LLMs cannot plan downstream tool calls or extract fields reliably.
Numeric parameters lack min/max bounds. max_age_hours, within_hours, tail_lines have no documented constraints. LLMs may pass absurd values (negative hours, 0 lines) causing silent failures or unexpected behavior.
No tool annotations despite all tools being read-only. readOnlyHint should be set on all 7 tools to signal to agents that these are safe to call without side effects.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 65 | 2026-07-28+ | v2 |
Find functions marked TODO that still return a hardcoded value. Catches the reporting function that reads its input file, ignores it, and returns the constant it was seeded with during development.
List scheduled jobs (launchd on macOS, systemd on Linux) with exit status.
Error handling is absent. Tools do not validate inputs, return actionable error messages, or guide recovery. A malformed URL in check_endpoint or missing directory in check_log_freshness will likely raise an unhandled exception.
Response fields not optimized for chaining. list_jobs returns job metadata but does not document which fields downstream tools (e.g., audit) expect. Unclear if 'label' is the canonical identifier.