Loading ToolBench
A Node.js service that monitors security vulnerabilities in real-time and sends alerts to Slack, Microsoft Teams and email.
Atalaia demonstrates solid tool design with clear naming conventions (verb_noun pattern), comprehensive parameter schemas with enums and constraints, and detailed descriptions. All 11 tools follow consistent patterns. However, output schemas are not explicitly documented in the source, and error handling guidance is minimal. Parameter descriptions are strong (averaging 80+ chars), but the server lacks recovery hints and actionable error messages. Tool composition is excellent, each tool has a single responsibility and clear chaining via IDs (cveId, idOrUrl, id). The explain_vulnerability tool correctly marks itself as WRITE, showing awareness of state mutation. Pagination is well-implemented across list tools (limit, offset, sort, order). Missing: explicit output schema documentation and error recovery patterns.
A markdown report of the current state: the fleet's risk, the open vulnerabilities, the repositories that are most exposed, and the owners responsible for them.
Ask the model configured in Atalaia to write the plain-English explanation for a CVE and store it. Use it when get_vulnerability reports explained=false. Fails with the provider's own reason when no model is configured or reachable.
Everything known about one owner: the full record and the assignments that say which technologies and repositories they are responsible for.
Everything known about one repository: the full record, its dependencies, the vulnerabilities that reach it, and who owns it.
Fleet statistics: how many repositories, how many vulnerabilities, how many open, how many critical.
Everything known about one CVE: the full record including the stored explanation, a timeline reconstructed from its timestamps, which monitored repositories it reaches and through which dependency, and who owns them.
Output schemas not documented in source code. LLMs cannot plan downstream calls or extract fields without knowing response structure (e.g., what fields does get_vulnerability return? Is it a flat object or nested?). This forces agents to guess or make exploratory calls.
Error handling lacks recovery guidance. Tools do not document what errors are retryable, what the LLM should do on failure, or what invalid inputs look like. E.g., explain_vulnerability says 'Fails with the provider's own reason' but does not guide the agent on next steps.
get_stats and build_report lack detail in descriptions. get_stats is only 'Fleet statistics: how many repositories...' (60 chars), below the 10 - 1024 char baseline. build_report is similarly terse. These tools need context on when to call them and what they return.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 68 | 2026-07-28+ | v2 |
The dependencies a repository declares. Filters are combined with AND. Outdated means the latest version is newer than what the manifest pins.
The system owners — the people responsible for technologies and repositories. An owner is a name and the channels that reach them, without saying where any of them go.
The repositories this fleet monitors. Filters are combined with AND. Returns a summary per repository — call get_repository for the full record.
Search the collected vulnerabilities. Filters are combined with AND. relevance=affecting narrows to CVEs that name something this fleet actually depends on; relevance=infrastructure narrows to container images and CI actions. Returns a summary per CVE — call get_vulnerability for the description and correlation.
Which still-open CVEs list any of these technologies as affected. The direct question to ask about a stack Atalaia does not monitor as a repository yet. Matches the affected-technology list exactly, not the prose — use list_vulnerabilities with search for a substring of the title.
Parameter descriptions in list_* tools could specify what 'sort' and 'order' do together. E.g., 'sort by first_seen_at in ascending order' is clearer than separate enum descriptions. Current descriptions assume LLM understands the interaction.
No confirmation or dry-run pattern for explain_vulnerability, which is a WRITE operation. If the LLM calls this on a sensitive CVE, it immediately generates and stores an explanation. A confirm_before_execute pattern would prevent accidental explanations.