Loading ToolBench
Supply-chain incident response console on HydraDB. MCP server for coding agents to query a dependency graph of watched GitHub repositories, their package-lock.json history, npm versions, OSV advisories and maintainers.
Reachable defines 12 tools with clear domain purpose (supply-chain incident analysis). All tools have descriptions (194 - 300 chars, within baseline 34 - 392 range) and input schemas with type declarations. However, schemas lack depth: parameter descriptions are minimal (10 - 30 chars vs. baseline 72 chars), no enums for constrained inputs (e.g., advisory ID format), no output schemas documented, and error handling is absent. Tool names are domain-specific but not verb-prefixed (e.g., 'exposed_services' vs. 'get_exposed_services'). Composition is sound, tools chain via advisory/service/package IDs, but descriptions do not guide LLM selection or explain when to use one over another. STDIO transport caps protocol readiness at 50.
Which versions the advisory affects, first affected, publish time, whether npm removed them and the installable window (Q2).
Run a read-only OpenCypher statement against the graph (MATCH/CALL/RETURN only; the worker refuses writes). Schema: (Service)-[:HAS_LOCKFILE]->(Lockfile)-[:RESOLVED]->(Version)-[:VERSION_OF]->(Package); (Version)-[:DEPENDS_ON]->(Version); (Advisory)-[:AFFECTS {live_from, live_to}]->(Version); (Maintainer)-[:MAINTAINS]->(Package). Ids are integers; match on the string `key` property (pkg:npm/debug, svc:owner/repo, MAL-2025-46974).
Which watched services resolved a version affected by this advisory (Q1), with the lockfile sha, commit time and the dependency it was pulled in through. Optional service (owner/repo) narrows to one.
Public GitHub repos whose package-lock.json pins an affected version today (GitHub code search on the tarball name). Candidates to watch — not verdicts.
Progress and log of an ingest job started by watch_repository.
Tool names lack action verbs. 'exposed_services', 'resolved_while_live', 'affected_versions' are noun-heavy; LLMs infer intent from verb-prefixed names like 'get_exposed_services', 'list_affected_versions'. Baseline: 90% of A+ tools start with action verb.
Parameter descriptions are minimal (10 - 30 chars). Baseline is 72 chars. E.g., 'advisory' param in exposed_services says 'Advisory ID (MAL-2025-46974, GHSA-xxxx-xxxx-xxxx, or CVE-2025-1234)', good examples but no constraint format stated. Missing: 'Must match pattern MAL-\d+, GHSA-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}, or CVE-\d{4}-\d{4,}.'
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 51 | <=2025-11-25 | v2 |
The watched repositories: key, lockfile count and latest ingested commit.
Maintainers of the affected packages and the other packages they publish, with which watched services resolve those today (Q4 — exposure if the next package is compromised).
Which services committed a lockfile pinning the bad version while it was still installable on npm (Q3). Evidence classes: in_window (commit inside the live window) and pinned_removed (pins a version npm has since erased).
Package names within one edit of the given npm package in the ingested graph (Q5). Distance and kind are facts; 'typosquat' is a hypothesis.
Start watching a GitHub repository (owner/repo or URL): ingests its package-lock.json history into the graph as a background job. Returns the job id; poll with job_status. Re-running for a repository already being ingested returns the running job rather than a second one.
Which watched services pin an exact npm package version in any ingested lockfile.
The dependency chain(s) through which a watched service (owner/repo) resolves an npm package, per lockfile — the 'why is this in my tree' proof path.
No output schemas documented. Tools return complex nested structures (e.g., exposed_services returns lockfile sha, commit time, dependency chain) but LLMs cannot plan downstream calls without knowing field names and types. Baseline: 100% of A+ tools have documented return types.
No error handling guidance. Tools lack descriptions of failure modes, recovery steps, or error classification (retryable vs. user-fixable vs. fatal). E.g., watch_repository may fail if repo is private or deleted, no guidance on what to do next.
cypher tool accepts arbitrary OpenCypher queries with minimal validation hint. Description says 'read-only' but LLMs may attempt writes if not strictly enforced server-side. No mention of query timeout, result limit, or injection prevention.