Loading ToolBench
Strong foundation with 16 well-defined tools, all with explicit schemas and descriptions. Tool naming follows verb_noun convention (get_, list_, create_, delete_, set_). All tools have ToolAnnotations (readOnlyHint, destructiveHint, idempotentHint). Descriptions are clear and action-oriented (avg ~120 chars). Input schemas use Pydantic models with proper type constraints (minLength, maxLength, minItems, maxItems). Key gaps: parameter descriptions are absent from schema definitions (only visible in Pydantic models, not in JSON Schema output); output schemas are not documented; error handling lacks recovery guidance; no pagination support for list tools despite potentially large result sets.
Create a group-restricted OIDC client, attach the exact requested groups, verify all security-relevant postconditions, and best-effort delete the client if verification fails.
Add a confidential OIDC client secret and write its one-time value directly to a new exclusive mode-0600 file. Existing secrets remain valid until explicitly deleted; the clear secret is never returned.
Delete one OIDC client only when its current name matches expected_name and confirm is true.
Get one Pocket ID OIDC client with secret-bearing fields omitted.
List Pocket ID OIDC clients with secret-bearing fields omitted.
Delete one light/dark OIDC client logo after explicit confirmation and verify it is gone.
List tools lack pagination support. oidc_client_list, user_group_list, user_list, and oidc_client_secret_list accept only a search parameter but do not return total_count, next_cursor, or accept limit/offset. Large result sets will blow context windows.
Output schemas are not documented. Tool descriptions state what is returned (e.g. 'with secret-bearing fields omitted') but the actual response structure is not visible in the tool definition. LLMs cannot plan downstream calls without knowing which fields are present.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | C | 63 | 2026-07-28+ | v2 |
Upload or replace one light/dark OIDC client logo from the configured bounded local logo asset directory.
Delete exactly one OIDC client secret by ID after explicit confirmation and verify it is gone.
List non-secret metadata for an OIDC client's secrets, including IDs, prefixes, timestamps and active state.
Replace the exact allowed-group set of an OIDC client that is already group restricted and verify the postcondition.
Return the bounded public OIDC discovery contract for the configured Pocket ID issuer.
Check Pocket ID discovery and authenticated API access without changing state.
Get one Pocket ID user with a bounded non-secret field set.
Get one Pocket ID user group without changing Pocket ID state.
List Pocket ID user groups without changing Pocket ID state.
List Pocket ID users with a bounded non-secret field set.
Parameter descriptions are missing from JSON Schema output. Pydantic models define constraints (minLength, maxLength) but the schema does not include 'description' fields for parameters like 'search', 'id', 'callback_urls', etc. LLMs cannot infer parameter semantics from names alone.
Error handling lacks recovery guidance. The call_tool handler validates inputs and calls service methods, but no error responses are visible in the code snippet. Errors should categorize as retryable/user-fixable/fatal and suggest next steps (e.g. 'Client not found. Try oidc_client_list() to find valid IDs.').
Destructive operations (delete, set_allowed_groups, logo_upload) require explicit confirmation via a 'confirm' boolean parameter, which is good. However, the tool descriptions do not explain the confirmation requirement or warn that these operations are irreversible. Add explicit language: 'This operation is irreversible. Set confirm=true only after verifying the action.'