Loading ToolBench
InfoSec & Cybersecurity control framework evidence knowledge base. Returns exact control citations, version lineage, cross-framework mapping edges, provenance, and explicit gaps over MCP.
Compliary registers 5 tools with descriptions and tool annotations (ReadOnlyHint, OpenWorldHint, DestructiveHint). However, critical gaps limit production readiness: 2 of 5 tools (guide, corpus_status) have NO input schemas at all, violating the schema requirement. The remaining 3 tools show empty InputSchema objects (inputSchemaFor[T]() returns {"type":"object","properties":{}} with no parameter descriptions), failing the parameter-description baseline (100% of A+ tools have param descriptions). Descriptions are present but generic, most lack actionable detail on WHEN to use each tool or what dependencies exist. No output schemas are documented. Error handling is not visible in the provided code. Tool naming follows verb_noun convention (search, document, quality_gaps) but lacks clarity on mutual exclusivity and parameter relationships.
Live per-framework/version counts: documents, controls by kind, withdrawn, chunks, embeddings, mapping edges (resolved/unresolved), completeness, last-stage info.
Citation lookup: control body (verbatim past auth), mapping edges (both directions, resolved + unresolved with honest labels), version lineage, parent/children context. Default version = current; explicit version pin supported.
Playbook for using compliary's evidence tools: scope, citation forms per framework, version-pin semantics, mapping-edge semantics, gaps philosophy, query tips including the framework filter's recall advantage (~93% filtered vs ~86% unfiltered).
Unresolved mapping edges by target, curated-mapping drift (seeded operator edges not materialized — a stale mapedges stage), deferred docs (amendments, CAIQ), unrecognized manifest rows, body-quality caveats (PCI guidance interleave), abstain/eval floors.
Hybrid retrieval (dense + BM25, RRF-fused) over InfoSec control frameworks. Accepts framework and version_label filters, include_withdrawn flag, top_k, mode, detail. detail='compact' is the cheap discovery pass (strips content/context_prefix); detail='standard' (default) returns the full hit shape. Citation-shaped queries are pinned at score 1.0. Score-floor abstention returns an explicit gap notice.
quality_gaps, document, and search register empty InputSchema objects (no properties, no parameter descriptions). Violates baseline: 100% of A+ tools have parameter descriptions. LLMs cannot infer what parameters these tools accept or what they control.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields. Violates baseline: 100% of A+ tools have documented return types.
Descriptions lack actionable detail. 'Citation lookup' and 'Hybrid retrieval' are vague on WHEN to use each tool vs alternatives, dependencies, or prerequisites. Descriptions should answer: What does it do? When should the LLM call it instead of a similar tool? What does it return?
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 46 | 2026-07-28+ | v2 |
No error handling guidance visible. Tool handlers (handleGuide, handleDocument, etc.) are not shown. Cannot verify if errors tell LLMs what to do next or categorize as retryable/user-fixable/fatal.