Loading ToolBench
Pulls a framework + its controls + evidence requirements from the TheArtOfService Compliance API and emits a Markdown brief. Also provides course discovery tools for compliance training.
Six tools with clear, domain-specific descriptions (avg 156 chars). All have input schemas with types and descriptions. However, parameter descriptions lack actionable constraints (no enums, ranges, or format guidance). Output schemas are undocumented, LLMs cannot predict response structure for chaining. Error handling is minimal (no recovery guidance). Tool naming is verb-noun compliant but generic (search_*, list_*, get_*). No tool annotations (readOnlyHint, idempotentHint). Composition is sound, each tool has one responsibility, but responses lack chaining IDs needed for multi-step workflows.
Courses covering two or more standards TOGETHER. This is the query a plain product listing cannot answer. An organisation running SOC 2 and ISO 27001 at once does not want one course per standard, it wants the overlap.
Hit the agent-tier control endpoint (no auth required) which returns evidence_requirements inline. The /api/frameworks/controls/{code} endpoint is the licensing-tier sibling and requires a Bearer key.
Retrieve a compliance framework by name
List all controls for a given compliance framework
Every standard the catalogue covers, with a course count each.
Find courses by what the buyer is trying to do.
No output schemas documented. LLMs cannot predict response structure (fields, types, nesting) for downstream tool calls or data extraction. Breaks tool chaining.
Parameter descriptions lack actionable constraints. 'frameworks' param in courses_for_frameworks has no format guidance (comma-separated? quoted?). 'limit' params have no min/max bounds. LLMs guess at valid input.
No error handling guidance. API calls use raise_for_status() but return no recovery hints. If 'NIST SP 800-161' is misspelled, get_framework returns None silently, LLM has no hint to try search or list alternatives.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 65 | 2026-07-28+ | v2 |
No tool annotations. All tools are read-only (risk: READ_ONLY declared in metadata) but no readOnlyHint in schema. Agents cannot infer safety from tool definitions alone.
Responses lack chaining IDs. search_courses returns course objects but no framework_id or control_code to feed into get_control_detail. Agents cannot compose workflows without extra discovery calls.