Loading ToolBench
Model Context Protocol server for Apache Kafka — monitor and manage clusters, topics, and consumer groups with security modes and access-control flags.
Kafka server has strong naming (clear verb_noun conventions), good descriptions on most tools (avg ~140 chars), but inconsistent schema quality and limited error guidance. Tool definitions are explicitly registered in src/server.ts with annotations. Input schemas are present and typed for all 12 tools, but some parameter descriptions are minimal (1-5 words). Output schemas are undocumented, callers must infer return types from tool names and descriptions. Error handling returns isError flag but lacks actionable recovery guidance or error classification. Security features are well-designed (mode-based access control, audit logging, confirmation flows) but not exposed in tool definitions. No composition issues detected, tools are granular and follow single-responsibility principle. Average parameter count is 1-2, reducing cognitive load.
Update one or more topic-level configuration entries.
Describe the Kafka cluster: brokers, controller, and cluster id.
Increase a topic's partition count (cannot decrease). Affects key-based ordering.
Create a topic with a partition count, replication factor, and optional configs.
Delete a consumer group (must have no active members). Requires admin mode AND KAFKA_ALLOW_DELETE=true.
Permanently delete a topic and its data. Requires admin mode AND KAFKA_ALLOW_DELETE=true. Irreversible.
Describe a consumer group's state and compute per-partition and total lag across its topics.
Output schemas are completely undocumented. Callers must infer return types from tool names and descriptions. LLMs cannot plan subsequent calls reliably without knowing field names, types, and cardinality.
Error handling returns only isError flag and text message. No error classification (retryable vs user-fixable vs fatal), no recovery guidance, no contextual hints for tool selection on failure.
Parameter 'configs' in create_topic and alter_topic_config accepts free-form object with no schema validation or documentation of allowed keys (e.g., 'retention.ms', 'segment.ms'). LLMs cannot discover valid config entries without trial-and-error.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | B | 75 | 2026-07-28+ | v2 |
Partitions, replicas, in-sync replicas, and non-default configs for a topic.
List consumer groups and their protocol types.
List topic names. Topics outside the allowlist are filtered out.
Reset a consumer group's committed offsets for a topic to the earliest or latest offset. The group must have no active members. Affects redelivery — use with care.
Earliest and latest offsets per partition for a topic (message backlog view).
Permission/security policies enforced server-side (mode, allowlist, protected topics) are not reflected in tool descriptions or parameter constraints. LLMs cannot predict failures due to access control without calling the tool and hitting PolicyError.
Tools like reset_consumer_group_offsets and delete_* have irreversible consequences but lack confirmation/dry-run mechanism in parameter definition. Elicitation is implemented server-side but not advertised in tool schema, so client cannot warn user proactively.
List tools (list_topics, list_consumer_groups) have no pagination parameters (limit, offset, cursor). If a cluster has hundreds of topics/groups, response will be truncated or massive, wasting tokens or risking context overflow.