Apache Spark Interview Questions and Answers (2026)

RDD is the low-level abstraction — a distributed collection of JVM objects with functional transformations, no schema, and no optimizer support. DataFrame adds a schema and runs through Catalyst and Tungsten but is untyped at compile time; it is literally Dataset[Row]. Dataset[T], available in Scala and Java, adds compile-time type safety over case classes while keeping Catalyst optimization. Default to the DataFrame/Dataset API and drop to RDDs only for needs like custom partitioning.

Transformations like map, filter, and join build up a logical plan without executing anything; actions like count, collect, and write trigger a job. Laziness lets Spark see the whole plan before running it, so Catalyst can push filters down, prune columns, and pipeline narrow operations into a single stage. A common follow-up: calling the same action twice recomputes the entire lineage unless you cache the intermediate result.

A narrow transformation (map, filter, union) needs only one parent partition per output partition, so it can be pipelined in memory. A wide transformation (groupByKey, a join without co-partitioning, distinct, repartition) needs data from many parent partitions, forcing a shuffle. The DAGScheduler cuts stage boundaries exactly at shuffle dependencies — every stage is a run of narrow operations between shuffles.

A shuffle redistributes data across the cluster by key: map-side tasks write sorted, partitioned blocks to local disk, and reduce-side tasks fetch their blocks over the network. That means serialization, disk I/O, and network transfer all at once, plus a hard stage boundary. The shuffle partition count defaults to spark.sql.shuffle.partitions=200, which is rarely right for your data size until AQE coalesces it at runtime.

reduceByKey applies the combining function on the map side first, so each partition sends one partially aggregated value per key across the network. groupByKey ships every raw value, inflating shuffle size and risking OOM on hot keys. The same principle carries into DataFrames — aggregations do partial aggregation automatically, which is one more reason to prefer the SQL API over raw RDDs.

Catalyst runs four phases: analysis resolves columns and tables against the catalog; logical optimization applies rule-based rewrites like predicate pushdown, column pruning, and constant folding; physical planning generates candidate plans and selects one, including the join strategy choice between broadcast hash and sort-merge using size statistics; finally, whole-stage code generation compiles the chosen plan into JVM bytecode. This is why filter conditions written as column expressions beat opaque lambdas — Catalyst can only rewrite what it can inspect.

Tungsten is the execution layer: it stores rows in a compact binary UnsafeRow format managed off-heap, eliminating Java object overhead and most GC pressure, and lays data out for CPU cache efficiency. Whole-stage code generation fuses a chain of operators into a single generated function, removing virtual function calls and iterator overhead. In short, Catalyst decides what plan runs; Tungsten decides how efficiently it executes.

If one side's estimated size is under spark.sql.autoBroadcastJoinThreshold (10MB by default), Spark ships that entire table to every executor and joins without shuffling the large side. You force it with broadcast(df) or the /*+ BROADCAST */ hint when statistics are missing or wrong — common after filters whose selectivity Spark cannot estimate. Mention the failure modes: broadcasting something too big can OOM the driver, which collects the table first, and AQE can now convert a sort-merge join to broadcast at runtime once actual sizes are known.

persist() takes a StorageLevel: MEMORY_ONLY (the RDD default) drops partitions that don't fit and recomputes them from lineage; MEMORY_AND_DISK (the default for DataFrame cache()) spills to disk instead; serialized, DISK_ONLY, OFF_HEAP, and replicated (_2) variants trade CPU, space, and resilience. Caching is a mistake when data is used once, when recomputation is cheaper than storage, or when cached blocks squeeze the execution memory a shuffle needs, since both share the unified pool. Always unpersist() when done.

repartition(n) always performs a full shuffle and can increase or decrease partition count with even distribution; coalesce(n) is a narrow operation that merges existing partitions without a shuffle. The trap: coalesce collapses into the previous stage, so coalesce(1) before a write can force the entire upstream computation onto a single task. For 'write fewer output files', repartition or AQE partition coalescing is usually the safer answer.

AQE re-optimizes the plan at shuffle boundaries using real statistics from completed stages. Its three headline features are coalescing small shuffle partitions (fixing the shuffle.partitions=200 guess), converting sort-merge joins to broadcast joins when a side turns out to be small, and splitting skewed partitions in joins via spark.sql.adaptive.skewJoin.enabled. It has been on by default since Spark 3.2, so lead with AQE and present manual shuffle-partition tuning as the fallback.

With spark.dynamicAllocation.enabled=true, Spark requests executors as tasks queue up and releases them after executorIdleTimeout, bounded by minExecutors and maxExecutors. The catch is shuffle data: a released executor takes its shuffle files with it, so you need either the external shuffle service on YARN or spark.dynamicAllocation.shuffleTracking.enabled on Kubernetes, which keeps executors holding live shuffle data alive. It is the standard setup for shared clusters running jobs with bursty stages.

That signature is data skew — one shuffle partition holds a hot key. Confirm in the Spark UI by sorting tasks by duration and shuffle read size; a task reading many times the median is the culprit, and the key is often null or a default value. Fix in order: filter or separately handle null and junk keys, let AQE's skew-join splitting take it, broadcast the smaller side to remove the shuffle entirely, or salt — append a random suffix 0..N to the hot key on the large side, explode the small side N ways, and aggregate in two passes. Note that AQE splits skew only for joins, not aggregations, so salting still earns its place.

Reserve roughly 1 core and 1GB per node for the OS and daemons, then target about 5 cores per executor — the sweet spot for HDFS/S3 client throughput. That yields 3 executors per node at 5 cores each; dividing the remaining ~63GB by 3 gives ~21GB per executor, from which spark.executor.memoryOverhead (the larger of 384MB or 10%, more with off-heap use) must be subtracted, so set roughly 18–19GB of heap. Avoid both extremes: single-core executors lose broadcast sharing within the JVM, while one fat executor per node suffers long GC pauses and HDFS client contention.

Of executor heap minus a 300MB reserve, spark.memory.fraction (0.6) forms the unified pool, split between execution memory for shuffles, joins, sorts, and aggregations and storage memory for cached blocks by spark.memory.storageFraction (0.5) — but the boundary is soft: execution can evict cached blocks down to the storage floor, while storage can never evict execution. Real OOMs usually come from the other 40% — user objects, or one huge task after skew — or from memoryOverhead breaches (Netty buffers, off-heap allocations) where YARN or Kubernetes kills the container rather than throwing a Java OOM. Distinguishing a heap OOM from a container kill is the senior-level detail interviewers reward.

A Scala lambda is a black box to Catalyst: the optimizer cannot push predicates past it, prune columns it might touch, or fold it into generated code, so whole-stage codegen breaks at that operator. Worse, every row must be deserialized from Tungsten's binary UnsafeRow into your case class through the encoder and serialized back afterwards — per-row ser/de that column expressions avoid entirely. The practical rule: express logic in Column functions or Spark SQL wherever possible, and confine typed lambdas to logic that genuinely cannot be written relationally.