Loading ToolBench
Server has 4 tools with reasonable coverage of ATT&CK query patterns. Tool naming is verb-led and clear (query_*, search_*). Descriptions are present and moderately detailed (ranging 150-250 chars), meeting minimum bars but lacking LLM-optimized specificity in places. Input schemas are visible with type definitions and descriptions, but descriptions contain Chinese text mixed with English, which may confuse LLM tokenization. Outputs are mentioned in descriptions but output schemas are NOT formally documented. Error handling is present in code (HTTPException, logging) but recovery guidance is minimal. Parameter descriptions use Chinese primarily, creating localization friction. No tool annotations (readOnlyHint, etc.) despite all being read-only operations.
Query detection methods or data components associated with an ATT&CK technique ID. Returns the source (data component name) and description for each relevant detection.
Query the list of mitigations related to a specific ATT&CK technique ID. Returns ID, name, and description for each applicable mitigation.
Query ATT&CK technique details by exact technique ID or fuzzy technique name search. ID query returns full data for a single technique including ID, name, description, platforms, kill chain phases, references, and subtechniques. Name search returns a summary list of matching techniques with ID, name, and short description.
Query comprehensive details of ATT&CK techniques by exact ID or fuzzy name search. Returns full information for matching techniques, including ID, name, description, platforms, kill chain phases, references, subtechniques, and mitigations. ID query returns single technique; name search returns list with count.
Parameter descriptions are primarily in Chinese, creating tokenization and comprehension issues for LLMs trained primarily on English text. Example: 'technique_id' param in query_technique is described as '要查询的ATT&CK技术ID (例如 'T1059.001')。若提供，将优先进行ID查询。' Mixed-language descriptions reduce LLM confidence and increase token overhead.
Output schemas are NOT documented. All 4 tools describe return data in prose (descriptions mention 'returns dict' or 'returns list') but do not include formal JSON Schema definitions for outputs. LLMs cannot reliably infer the structure of responses without explicit schema.
No tool annotations present. All 4 tools are read-only queries (risk=READ_ONLY listed in metadata) but do not use readOnlyHint annotations. Per current spec (2026-07-28), tool annotations are a recommended pattern for clarity. This is a minor gap but impacts spec alignment.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | B | 70 | <=2025-11-25 | v2 |
Two tools (query_technique and search_technique_full) have overlapping responsibility. Both accept technique_id OR tech_name and both return technique details. query_technique returns summaries on name search; search_technique_full returns full details including mitigations. The distinction is subtle and not immediately clear from names.
Error messages in code (e.g., HTTPException with generic 'Query failed') do not follow recovery-guide pattern. When technique_id is not found, the error is '{"error": "未找到技术ID {technique_id}"}' without suggesting alternatives or next steps.
No pagination or result limiting for name searches. search_technique_full and query_technique both perform fuzzy name searches that could return hundreds of results. No pagination parameters (limit, offset, page) and no documented result caps.
LLMs tend to reuse example values literally. Should use enum constraints or format descriptions instead.