Loading ToolBench
MCP server for navigating and understanding Keycloak source code locally
This server has 20 tools with schemas and descriptions present for all tools, which is a solid foundation. However, there are significant issues: (1) Parameter descriptions are often generic or incomplete. For example, 'version' appears on 18 tools with identical boilerplate text, forcing LLMs to understand the version system rather than having it self-evident. (2) Output schemas are NOT documented, the code shows tools returning text via a handler function, but nowhere are the response field structures described. This forces LLMs to blindly extract data without knowing what fields to expect. (3) Several tool descriptions are vague about when to use them vs. similar tools (e.g., explain_implementation vs. get_class_source; grep_source vs. search_spi_definitions). (4) The 'compare_versions' tool has complex conditional logic (target='class' vs 'spi_scan') but parameter relationships are not clearly documented, the 'query' param is only required for one target mode, but this is buried in the description rather than enforced. (5) Tools touching a running Keycloak instance (keycloak_admin, analyze_logs, diagnose_user) do not document error handling or recovery paths if the instance is unavailable. (6) No tool hints the destructive/read-only nature except via the Risk field in the spec, tool annotations (readOnlyHint, destructiveHint) are not implemented. (7) Several tools (trace_authentication_flow, visualize_auth_flow) accept complex input (JSON snapshots, realm exports) without validation guidance. Overall: solid structural foundation, but descriptions lack LLM-optimization and output contracts are undocumented.
Register a Keycloak version from a local git branch or tag.
Analyze Keycloak logs to understand recent activity and errors.
Check Keycloak GitHub security advisories for CVEs affecting a version.
Compare Keycloak source across two versions. Use target='class' (default) with `query` to diff a specific class or interface. Use target='spi_scan' to scan well-known SPI interfaces (or a custom list) for breaking changes.
Connect to a running Keycloak dev instance and validate the connection.
Diagnose a Keycloak user's status, credentials, sessions, and login events.
Output schemas are completely undocumented. All tools return text via a handler function, but the response structure (fields, types, pagination) is never formally specified. This forces LLMs to parse unstructured text and guess field names for downstream tool chaining.
Tool annotations (readOnlyHint, destructiveHint, idempotentHint) are absent. The Risk field is set in the spec metadata, but tools do not declare these hints in their registration. This prevents LLMs from understanding operation safety without reading descriptions.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-22 | C | 66 | 2026-07-28+ | v2 |
| 2026-03-09 | F | 35 | - | v1 |
Primary tool for understanding Keycloak internals. Accepts natural language queries about features, classes, or flows. Orchestrates deep source analysis including class hierarchies, interface methods, SPI extension points, implementations, and dependencies. Examples: "How does authentication flow work?", "Explain ExecuteActionsActionTokenHandler", "What happens during password reset?", "RequiredActionProvider"
Find all classes that implement a given interface or extend a given class.
Get the full source code of a specific Java class. Auto-discovers file if not found at the given path.
Get the current configuration of the connected Keycloak dev instance.
List all SPI providers loaded in the running Keycloak instance, including custom extensions.
Full-text search across the Keycloak source code. Uses ripgrep with regex support.
Connect to a running Keycloak instance and perform admin queries.
List all registered Keycloak source versions available locally.
Search and list SPI definitions in META-INF/services files.
Trace an authentication flow in a running Keycloak instance with real-time source code annotations.
Trace what a Keycloak class depends on and what depends on it.
Analyze custom Keycloak SPI implementations for upgrade compatibility.
Validate that custom SPI providers are correctly registered and loadable in the running instance.
Visualize a Keycloak authentication flow as a Mermaid diagram.
Parameter relationships are undocumented. For example, 'compare_versions' has conditional logic where 'query' is required only when target='class', but this constraint is buried in text description and not enforced by schema validation.
Error handling and recovery guidance is missing. Tools that connect to a running Keycloak instance (keycloak_admin, analyze_logs, get_loaded_providers, trace_authentication_flow, validate_spi_registration, get_dev_instance_config, diagnose_user) do not document what happens if the connection fails, times out, or the instance is unavailable. No recovery instructions provided.
Generic boilerplate parameter descriptions. The 'version' parameter appears on 18 tools with identical text: 'Optional version name (e.g. "v24", "v26"). Uses default if omitted. See list_versions.' This is not LLM-optimized, LLMs read this for every tool and waste reasoning cycles. Consider a shared definition or richer context about version availability.
Input validation and error messages are not documented. Tools accepting complex inputs (trace_authentication_flow with JSON snapshot, visualize_auth_flow with realm export path) do not explain validation rules, format requirements, or error messages if invalid input is provided.
Tool disambiguation is incomplete. 'explain_implementation' vs 'get_class_source' have overlapping intent (both retrieve and explain code). 'grep_source' vs 'search_spi_definitions' overlap in search capability. Descriptions do not clearly state when to prefer one over another, forcing LLM reasoning overhead.
Result limits are not enforced or documented. 'grep_source' accepts maxResults with default 30 and max 100, but other tools (get_loaded_providers, list_versions, etc.) do not declare limits or pagination. Large result sets could exhaust context windows.
Tools requiring external connections lack timeout and state guarantees. 'connect_dev_instance' and tools operating on the dev instance do not document retry behavior, timeouts, or what happens if state is lost mid-flow (e.g., trace_authentication_flow with snapshot).