Spark Connect (gRPC) MCP Server — session SQL execution, versioned Transform submission/rerun against the shared Iceberg lakehouse catalog (scope=lakekeeper, explicit), catalog reads, OpenLineage config/status reporting, and a Wire-First KG ingest tool for the SparkApplication->SparkJob->Transform->TransformRun->DatasetVersion chain.
Spark MCP has 7 tools with basic naming and descriptions, but significant gaps in schema completeness and parameter documentation. Tool names follow verb_noun convention (spark_sql, spark_describe, spark_cancel), which is good. However, input schemas are visible but lack depth: most parameters have only type and description, with no enums, constraints, ranges, or validation guidance. Descriptions are present but brief (avg ~80 chars), missing context on when to use each tool and what happens on failure. No output schemas documented. Error handling is not evident in tool definitions. The spark_cancel tool accepts an optional 'tag' parameter but provides no guidance on valid tag formats or what happens if the tag doesn't exist. spark_submit_transform and spark_rerun_transform are write operations but lack confirmation/dry-run patterns. Overall, definitions are functional but lack the depth needed for confident LLM tool selection and error recovery.
Retrieve the current Spark application status and metadata.
Interrupt running operations on this session (all, or one tagged operation).
Schema + column info for one fully-qualified table.
List tables in a namespace of the Iceberg lakehouse catalog.
Rerun a previously submitted Transform.
Execute one SQL statement and return rows as plain dicts. Never degrades a connection failure or a query error to an empty result — distinguishes 'Spark Connect unreachable' from 'the query itself failed'.
Submit a versioned Transform against the shared Iceberg lakehouse catalog.
No output schemas documented for any tool. LLMs cannot plan downstream calls or extract required fields without knowing response structure.
Write operations (spark_submit_transform, spark_rerun_transform) lack confirmation/dry-run patterns. Agents cannot preview changes before irreversible execution.
Parameter descriptions lack constraints, ranges, and validation guidance. E.g., spark_sql row_limit has no min/max bounds; spark_cancel tag parameter has no format specification.
Inferred effective spec: <=2025-11-25.
| Scored | Grade | Overall | Spec posture | Rubric |
|---|---|---|---|---|
| 2026-09-23 | F | 46 | <=2025-11-25 | v2 |
No error handling guidance in tool definitions. LLMs cannot determine if failures are retryable, user-fixable, or fatal.
spark_list_lakekeeper_tables lacks pagination support. No limit, offset, or cursor parameters documented for potentially large result sets.