Loading ToolBench
Leash exposes 20 tools across identity, authorization, and audit domains. Most tools lack proper input schemas, 6 tools have empty input objects ({}), and internal service functions expose SQLAlchemy session and Pydantic model objects as parameters, which violates secret injection and type safety patterns. Descriptions are present but often generic or domain-specific without LLM-optimized guidance. Tool names follow verb_noun convention (good), but composition is poor: internal service functions (log_authorize_decision, create_audit_entry, dispatch_audit_event) are exposed as tools rather than encapsulated. No error handling guidance, no pagination metadata, no output schemas documented. This is a domain-specific authorization layer, not a general-purpose agent toolkit, tools like 'require_agent' and 'enforce_identity' are auth middleware, not agent-callable tools.
Rotate the server signing key-pair (admin-only). Archives the current key and generates a new pair. Existing JWTs remain valid via previous-key fallback. Old audit signatures also remain verifiable. Requires a valid admin JWT (type=cli, admin, or ops).
Create a signed JWT for an agent.
Append a new signed audit entry with a hash-chain link.
Serve the live web dashboard.
Send an audit event to all configured external sinks. Called from the audit service after every entry is committed. Both webhook and file sink are optional — if neither env var is set, this is a no-op.
Ensure the JWT belongs to the agent making the request. CLI-admin tokens (type='cli') are allowed to access any agent, enabling management operations like permissions inspection. Raises 403 if the token subject doesn't match the agent_id.
6 tools have empty input schemas ({}): health, key_info, admin_rotate_server_keys, metrics, dashboard, rotate_server_keys.
Internal service functions (log_authorize_decision, create_audit_entry, dispatch_audit_event) expose SQLAlchemy session and Pydantic model objects as parameters. These are not serializable, violate secret injection, and should not be exposed as agent-callable tools.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields (e.g., agent_id, token, signature). Violates response-shaper pattern.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 49 | 2026-07-28+ | v2 |
Retrieve audit entries with optional filters and pagination.
Compute aggregate stats across the entire audit log.
Return a SHA-256 hex digest for the canonical JSON of data.
Health check endpoint
Return server signing key metadata (age, permissions, rotation status). Requires admin JWT when LEASH_REQUIRE_AUTH_READ=true.
Auto-log an authorize decision to the audit trail. Called by the authorize route after every policy evaluation so that the audit table stays in sync with actual usage. This is fire-and-forget: errors are swallowed to avoid breaking the authorize response. Observe-mode denials are logged with policy_decision='observe_deny' so they are distinguishable from real allows and real denies.
Prometheus-compatible metrics endpoint. When LEASH_REQUIRE_AUTH_READ=true, requires a valid JWT in the Authorization header. Otherwise open (standard Prometheus scrape).
Like require_agent but returns None when no token is provided. Use for read-only endpoints that should work without auth but can accept a token for richer results.
Validate the Bearer token and return the decoded JWT payload. Raises 401 if the token is missing or invalid. The returned dict contains at least sub (agent_id) and name.
Like require_agent but enforces admin privileges when configured. When LEASH_POLICY_REQUIRE_ADMIN=true, only agents with an admin-class type (cli, admin, ops) or 'admin' in their name can manage policies. Permissive by default for single-user deployments.
Rotate the server signing key-pair with graceful transition. Archives the current key and generates a new pair. New JWTs and audit signatures use the new key. Existing JWTs signed by the old key remain verifiable via the previous public key (graceful fallback). Old audit signatures remain verifiable. Agents can re-register or call rotate at their leisure to get a JWT signed by the new key.
Produce an RSA-SHA256 hex signature over the canonical JSON of data.
Decode and verify a JWT against the current server key. Falls back to the previous server public key if the current key fails verification. This enables graceful server key rotation: existing JWTs remain valid during the transition window.
Verify an RSA-SHA256 hex signature against the server's public key. Falls back to the previous public key if the current key fails, so audit entries signed before a server key rotation remain verifiable.
Auth middleware tools (require_agent, optional_agent, enforce_identity, require_policy_admin) are not agent-callable tools, they are FastAPI dependencies. Exposing them as MCP tools conflates middleware with business logic.
No error handling guidance. Tools lack recovery hints (e.g., 'Invalid token. Call create_agent_token() to generate a new one'). Violates recovery-guide pattern.