Apache Spark Interview Questions and Answers (2026)

RDD is the low-level abstraction — a distributed collection of JVM objects with functional transformations, no schema, and no optimizer support. DataFrame adds a schema and runs through Catalyst and Tungsten but is untyped at compile time; it is literally Dataset[Row]. Dataset[T], available in Scala and Java, adds compile-time type safety over case classes while keeping Catalyst optimization. Default to the DataFrame/Dataset API and drop to RDDs only for needs like custom partitioning.

Transformations like map, filter, and join build up a logical plan without executing anything; actions like count, collect, and write trigger a job. Laziness lets Spark see the whole plan before running it, so Catalyst can push filters down, prune columns, and pipeline narrow operations into a single stage. A common follow-up: calling the same action twice recomputes the entire lineage unless you cache the intermediate result.

A narrow transformation (map, filter, union) needs only one parent partition per output partition, so it can be pipelined in memory. A wide transformation (groupByKey, a join without co-partitioning, distinct, repartition) needs data from many parent partitions, forcing a shuffle. The DAGScheduler cuts stage boundaries exactly at shuffle dependencies — every stage is a run of narrow operations between shuffles.

A shuffle redistributes data across the cluster by key: map-side tasks write sorted, partitioned blocks to local disk, and reduce-side tasks fetch their blocks over the network. That means serialization, disk I/O, and network transfer all at once, plus a hard stage boundary. The shuffle partition count defaults to spark.sql.shuffle.partitions=200, which is rarely right for your data size until AQE coalesces it at runtime.

reduceByKey applies the combining function on the map side first, so each partition sends one partially aggregated value per key across the network. groupByKey ships every raw value, inflating shuffle size and risking OOM on hot keys. The same principle carries into DataFrames — aggregations do partial aggregation automatically, which is one more reason to prefer the SQL API over raw RDDs.

Catalyst runs four phases: analysis resolves columns and tables against the catalog; logical optimization applies rule-based rewrites like predicate pushdown, column pruning, and constant folding; physical planning generates candidate plans and selects one, including the join strategy choice between broadcast hash and sort-merge using size statistics; finally, whole-stage code generation compiles the chosen plan into JVM bytecode. This is why filter conditions written as column expressions beat opaque lambdas — Catalyst can only rewrite what it can inspect.

Tungsten is the execution layer: it stores rows in a compact binary UnsafeRow format managed off-heap, eliminating Java object overhead and most GC pressure, and lays data out for CPU cache efficiency. Whole-stage code generation fuses a chain of operators into a single generated function, removing virtual function calls and iterator overhead. In short, Catalyst decides what plan runs; Tungsten decides how efficiently it executes.

persist() takes a StorageLevel: MEMORY_ONLY (the RDD default) drops partitions that don't fit and recomputes them from lineage; MEMORY_AND_DISK (the default for DataFrame cache()) spills to disk instead; serialized, DISK_ONLY, OFF_HEAP, and replicated (_2) variants trade CPU, space, and resilience. Caching is a mistake when data is used once, when recomputation is cheaper than storage, or when cached blocks squeeze the execution memory a shuffle needs, since both share the unified pool. Always unpersist() when done.

repartition(n) always performs a full shuffle and can increase or decrease partition count with even distribution; coalesce(n) is a narrow operation that merges existing partitions without a shuffle. The trap: coalesce collapses into the previous stage, so coalesce(1) before a write can force the entire upstream computation onto a single task. For 'write fewer output files', repartition or AQE partition coalescing is usually the safer answer.

That signature is data skew — one shuffle partition holds a hot key. Confirm in the Spark UI by sorting tasks by duration and shuffle read size; a task reading many times the median is the culprit, and the key is often null or a default value. Fix in order: filter or separately handle null and junk keys, let AQE's skew-join splitting take it, broadcast the smaller side to remove the shuffle entirely, or salt — append a random suffix 0..N to the hot key on the large side, explode the small side N ways, and aggregate in two passes. Note that AQE splits skew only for joins, not aggregations, so salting still earns its place.

A Scala lambda is a black box to Catalyst: the optimizer cannot push predicates past it, prune columns it might touch, or fold it into generated code, so whole-stage codegen breaks at that operator. Worse, every row must be deserialized from Tungsten's binary UnsafeRow into your case class through the encoder and serialized back afterwards — per-row ser/de that column expressions avoid entirely. The practical rule: express logic in Column functions or Spark SQL wherever possible, and confine typed lambdas to logic that genuinely cannot be written relationally.