Loading ToolBench
Curated threat-actor context for an indicator, from Google Threat Intelligence
Strong tool definitions with complete schemas and descriptions. All 12 tools have clear verb-noun naming (search_*, get_*, list_*, lookup_*) and detailed descriptions (avg 180 chars). Input schemas are well-formed with type constraints, enums, and parameter descriptions. Pagination support (limit 1-40) is consistent. However, output schemas are not documented in the source code, LLMs cannot see what fields to expect from responses. Error handling descriptions are minimal; tools lack recovery guidance. No tool annotations (readOnlyHint, destructiveHint) despite all being read-only operations.
The result-cache state: entry count, total size, oldest and newest write, and the configured TTLs. TTLs are applied at read time, so entries older than the TTL are already dead weight.
Sandbox behaviour summary for one file hash (MD5/SHA1/SHA256): an index of sections (dns_lookups, http_requests, processes, ...) with item counts, or one section expanded with its items. Sections are named by the sandbox vendor; the index is stable across queries.
One LiveHunt ruleset: id, name, enabled state, and the Yara rules text. The rules are returned as-is from upstream; syntax errors are upstream's.
Pivot from an indicator along one relationship: infrastructure it touches (contacted_domains, contacted_ips, contacted_urls), files around it (communicating_files, downloaded_files, dropped_files), embedded IOCs, or the curated threats naming it (associations, campaigns, malware_families, related_threat_actors, reports).
Curated report for one collection id (threat-actor--<uuid>, report--<hash>, malpedia_..., ...): description, aliases, targets, sources, dates. The description is capped at 6000 chars with the drop accounted; raise or lift the cap with description_max when the tail matters.
Output schemas not documented. LLMs cannot see what fields responses contain, forcing them to guess downstream field names and risk broken tool chains.
No tool annotations (readOnlyHint, destructiveHint, idempotentHint). All 12 tools are read-only but lack explicit annotation, preventing agents from optimizing retry logic and caching.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 64 | 2026-07-28+ | v2 |
ATT&CK tactics and techniques tree for one collection id: the techniques the threat actor uses, grouped by tactic. Techniques carry ids for pivots to get_threat_related.
Pivot from a collection to what it relates to: its IOCs (domains, files, ip_addresses, urls), its actors and families, its ATT&CK techniques, its detection rules (hunting_rulesets). Items carry names where upstream has them; ids double as input to get_threat / lookup_ioc.
The full reference: every tool, every parameter, every relationship name, the result schema, and the error-recovery table. Returned as Markdown rather than JSON.
Your LiveHunt rulesets: id, name, enabled state. Ids are input to get_hunting_ruleset. An account with no rulesets is a valid answer.
Actor context for one indicator — MD5/SHA1/SHA256 hash, IP, domain, or URL, detected from its shape: Google's own gti_assessment plus the curated threats (actors, campaigns, families, reports) associated with it. No associations is a valid answer. incomplete:true means the association expansion failed — treat that as unanswered, never as clean. The default answer is deliberately trimmed: per-engine verdicts, passive DNS, reputation and URL behaviour belong to malware-lookup, rdns-lookup, abuse-lookup and urlscan-lookup; full:true opts into GTI's whole report as a second opinion.
Search the IOC corpus with GTI intelligence syntax: free-form query with entity type and percentile filters (e.g. "wannacry entity:file p:60+"). Returns compact summaries with ids for lookup_ioc / get_ioc_related. retrieved/more/total_upstream account for what this page holds versus what upstream has.
Search the curated threat catalogue (collections): threat actors, malware families, campaigns, reports, software toolkits, vulnerabilities. Returns compact summaries with ids for get_threat / get_threat_related. Give a query, a collection_type, or both. retrieved/more/total_upstream account for what this page holds versus what upstream has.
Error handling lacks recovery guidance. Tools document error conditions but do not tell LLMs what to do next (retry, ask user, call alternative tool).
Parameter 'relationship' and 'relationship_other' in get_threat_related and get_ioc_related are mutually exclusive but lack explicit documentation of this constraint in descriptions.
cache_status and get_usage are utility/introspection tools with minimal descriptions (70 chars). Descriptions should explain when and why an agent would call them.