PySpark Interview Questions and Answers for Data Engineers

Spark is a distributed, in-memory engine for large-scale data processing. It parallelizes work across a cluster and keeps data in memory between steps, making it much faster than disk-based MapReduce for big data and iterative workloads.

RDDs are the low-level distributed collection with no schema. DataFrames add a named schema and are optimized by the Catalyst engine. Datasets (Scala/Java) add compile-time type safety. In PySpark you mostly use DataFrames for performance and simplicity.

Transformations (select, filter, join, groupBy) are lazy and build a plan without running. Actions (count, collect, write, show) trigger execution of that plan. Nothing computes until an action is called.

Spark records transformations as a logical plan and only executes when an action is called. This lets the Catalyst optimizer combine and reorder operations for efficiency, such as pushing filters down before joins.

Narrow transformations (map, filter) need data only from one partition and require no data movement. Wide transformations (groupBy, join, distinct) require a shuffle that moves data across partitions and is the main performance cost.

A shuffle redistributes data across partitions and executors for operations like joins and aggregations. It's expensive because it writes to disk and moves data over the network, so minimizing shuffles is key to performance.

repartition(n) reshuffles data into a new number of partitions and can balance skew. coalesce(n) only reduces partitions and avoids a full shuffle by merging existing ones, so it's cheaper for shrinking partition count after a filter.

A broadcast join sends a small table to every executor so the large table can be joined locally without shuffling it. Use it when one side is small enough to fit in memory; it avoids the costly shuffle of a standard sort-merge join.

Both store a DataFrame for reuse to avoid recomputation. cache() uses the default storage level (memory, spilling to disk). persist() lets you choose the level, such as memory-only or disk-only. Use them when a DataFrame is reused multiple times.

Detect skew from long-tail tasks in the Spark UI, then mitigate by salting the skewed key, broadcasting the small side, enabling Adaptive Query Execution's skew-join handling, or pre-aggregating before the join.

Python UDFs run row by row outside the JVM, adding serialization overhead and blocking Catalyst optimizations. Prefer built-in Spark SQL functions; if a UDF is unavoidable, use Pandas (vectorized) UDFs, which are much faster.

Read with spark.read.parquet(path) and write with df.write.partitionBy('date').parquet(path). Partitioning by a commonly filtered column enables partition pruning, so queries scan only the relevant folders.

Columnar formats like Parquet or ORC are preferred because they support column pruning, predicate pushdown, and compression, which drastically reduce I/O compared with row formats like CSV or JSON.

Catalyst is Spark SQL's query optimizer. It transforms the logical plan through rule- and cost-based optimizations (predicate pushdown, column pruning, join reordering) into an efficient physical plan, which is why DataFrames outperform raw RDDs.

The driver runs the main program, builds the DAG, and schedules work. Executors are worker processes that run tasks on partitions and hold cached data. Collecting too much data to the driver can cause out-of-memory errors.

Check the Spark UI for skew, large shuffles, and spills; review input sizes and partition counts; reduce data movement with broadcast joins or better partitioning; select only needed columns; and if needed raise executor memory or enable AQE.

AQE (Spark 3+) re-optimizes the plan at runtime using real statistics — coalescing shuffle partitions, switching join strategies, and handling skew automatically. Enabling it often improves performance without code changes.

Use a window: Window.partitionBy(key).orderBy(col('timestamp').desc()), assign row_number(), and filter where it equals 1. This keeps the most recent record per key without a costly self-join.

groupBy().agg() collapses rows into one per group. A window aggregation over a Window spec computes the aggregate while keeping every original row, which is needed for running totals, rankings, and per-row comparisons.