Loading ToolBench
Self-hosted AI context enforcement across LLM, MCP, vLLM, Ollama, and agent traffic — a stable, zero-dependency security gateway.
Haechi exposes 17 tools across auth, audit, crypto, and core protection domains. All tools have descriptions (10-200 chars, baseline 194) and input schemas with typed parameters. However, output schemas are entirely undocumented, no tool declares what it returns, forcing LLMs to infer structure. Parameter descriptions are present but often generic ('Crypto provider with hmac() method' appears 4 times verbatim). Tool names follow verb_noun convention (readAuthStore, validateLabels, addToken) but lack clarity on side effects, addToken, revokeToken, and createHaechi modify state but descriptions do not explicitly state this. Error handling is absent from all tool definitions. No tool declares permissions, audit trails, or recovery guidance. Composition is reasonable (single responsibility per tool) but chaining is unclear, createHaechi accepts 'tokenVault' as optional object but no tool documents how to construct one.
Add a new authentication token with type, scopes, and labels to the auth store
Conformance test suite for authProvider implementations to verify security behaviors
Build a PII-safe identity object from external auth provider claims
Build a PII-safe identity object from a token record using cryptoProvider
Create core-owned audit sink with sha256 hash-chain, sanitization, and anchor stream
Create a bearer token authentication provider that validates tokens against a store
Create a file-backed audit store implementing exclusive read-previous + persist primitive
Create core Haechi protection engine with filter, policy, crypto, and audit integration
Output schemas completely undocumented. No tool declares return type, fields, or structure. LLMs cannot plan downstream calls or extract required data.
State-modifying tools (addToken, revokeToken, createHaechi) lack explicit 'modifies state' language in descriptions. Agents cannot determine if calls are safe to retry.
Generic parameter descriptions repeated verbatim (e.g. 'Crypto provider with hmac() method' in 4 tools). Lacks context on when/why to use each tool or dependencies between them.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | D | 52 | 2026-07-28+ | v2 |
Create file-backed JSONL audit sink with optional anchor stream (back-compat wrapper)
Create AES-256-GCM local crypto provider with nonce budget enforcement and key rotation
List all authentication tokens from the store
Read and summarize audit JSONL file, returning event counts, detections, and breakdowns
Read authentication store from a file path, returning version and tokens array
Revoke an authentication token by ID
Recursively sanitize audit values by removing forbidden keys (PII-safe)
Validate authentication labels against allowed keys and value constraints
Verify sha256 hash-chain integrity of audit JSONL with optional anchor stream
No error handling guidance. Tools accept complex objects (filterEngine, policyEngine, auditSink) but do not document validation, required methods, or failure modes.
No permission declarations or audit trail guidance. Tools that modify auth state (addToken, revokeToken) do not specify required scopes or logging expectations.