PySpark Interview Questions and Answers for Data Engineers

Spark is a distributed compute engine for large-scale data processing. A driver program builds a DAG of operations, an optimizer plans them, and executors run tasks in parallel across partitions of the data. Its speed over classic MapReduce comes from keeping intermediate data in memory across stages and pipelining operations instead of writing to disk between every step, and it unifies batch, SQL, streaming, and ML in one API. Nuance interviewers probe: 'in-memory' does not mean the dataset must fit in RAM — Spark spills to disk and only caches what you ask it to. Also, Spark is overkill for data that fits on one machine; pandas or DuckDB beat it there because Spark pays JVM, scheduling, and shuffle overhead.

An RDD is the low-level distributed collection: no schema, just opaque objects, so Spark cannot optimize your logic. A DataFrame adds a named schema, letting the Catalyst optimizer rewrite the plan (predicate pushdown, column pruning) and generate efficient JVM code. A Dataset adds compile-time type safety, which exists only in Scala/Java — in PySpark, DataFrame is the API and there is no typed Dataset. The practical point interviewers want: PySpark DataFrame operations execute inside the JVM, while RDD lambdas must pickle every element out to Python worker processes and back, which is dramatically slower. Drop to RDDs only when you need per-record control that built-in SQL functions genuinely cannot express.

Transformations (select, filter, join, groupBy, withColumn) are lazy: they only add nodes to a logical plan. Actions (count, collect, show, write, take) submit that plan as a job and actually compute. Two follow-ups matter. First, each action re-executes the whole lineage from the source unless you cache — calling df.count() and then df.write() reads the input twice. Second, some calls look alike but differ: printSchema() touches only metadata, while show() runs a real (partial) job. The classic trap is collect(), which pulls the entire dataset into the driver's memory and is the most common cause of driver OOM; prefer take(n), show(), or writing results out instead.

Spark records transformations as a logical plan and defers execution until an action forces it. The payoff is optimization across the whole plan: Catalyst can push filters below joins, prune unread columns before scanning Parquet, and collapse consecutive projections — none of which is possible if each line executed eagerly. The trap interviewers probe is debugging: an error in a transformation (bad column name, cast failure, malformed data) surfaces only at the action, often with a stack trace far from the offending line, so a failure at df.write() may actually be a bug in a filter defined 50 lines earlier. Use df.explain() to inspect the plan without running it, and a small limit()-based action to smoke-test logic early.

Narrow transformations (map, filter, withColumn, union) compute each output partition from a single input partition, so tasks pipeline without moving data. Wide transformations (groupBy, join, distinct, orderBy, repartition) need rows with the same key gathered together, forcing a shuffle across the network. The distinction matters because shuffles are the stage boundaries in the DAG and dominate job cost. The trap: several innocent-looking calls are wide — dropDuplicates shuffles like a groupBy, orderBy performs a full range-partitioned sort, and coalesce, while avoiding a shuffle, can silently reduce upstream parallelism. Asked how to speed up a job, 'remove or shrink shuffles' is the expected first instinct.

A shuffle repartitions data by key so rows that must meet — same join key, same group — land in the same task. Mechanically, every upstream task writes shuffle files to local disk, then every downstream task fetches its slice from each of them over the network: serialization, disk I/O, and an all-to-all network exchange, which is why it dominates job cost. Shuffles also define stage boundaries and are where skew and spill show up in the Spark UI. Two tunables interviewers expect you to know: spark.sql.shuffle.partitions (long defaulted to 200, which suits almost no real data size) and AQE, which coalesces small post-shuffle partitions at runtime. Reduce shuffles with broadcast joins, pre-aggregation, and partitioned storage.

repartition(n) triggers a full shuffle to produce n evenly sized partitions — use it to increase parallelism or fix imbalance, optionally by column (repartition('key')) to co-locate data before a write. coalesce(n) only merges existing partitions, avoiding a shuffle, so it is the cheap way to shrink partition count after a heavy filter; it can never increase the count. The classic trap: coalesce is a narrow dependency that propagates upstream — df.coalesce(1).write makes the entire preceding stage run as a single task, not just the final write. If you need one output file without killing parallelism, repartition(1) keeps the upstream work parallel and pays one shuffle at the end, which is usually the better trade.

In a broadcast join, Spark ships the small table in full to every executor, so the large table is joined locally, map-side, and never shuffles — eliminating the exchange both sides of a sort-merge join would pay. Spark broadcasts automatically when it estimates a side below spark.sql.autoBroadcastJoinThreshold (10MB by default), and you can force it with broadcast(df) or a BROADCAST hint. Traps to name: the size estimate comes from statistics and can be badly wrong after filters or on fresh tables, and broadcasting a too-large table OOMs both the driver (which materializes it first) and the executors. Rule of thumb: small dimension tables yes; anything approaching executor memory, absolutely not.

Both mark a DataFrame for materialization and reuse so later actions skip recomputing the lineage. cache() is just persist() with the default storage level — MEMORY_AND_DISK for DataFrames — while persist() lets you choose: MEMORY_ONLY, DISK_ONLY, serialized or replicated variants. Points interviewers probe: caching is lazy, so nothing is stored until the first action materializes it; cache only when a DataFrame feeds multiple actions or iterative logic, because cached blocks compete with execution memory and can be evicted; and call unpersist() when done. Note the subtlety that MEMORY_ONLY drops partitions that don't fit and silently recomputes them, whereas MEMORY_AND_DISK spills — so memory-only is not automatically faster.

First prove it in the Spark UI: one or two tasks in a stage run far longer and read far more data than the rest — a hot key, not an undersized cluster. Then, in order of preference: broadcast the small side so the skewed key never shuffles; enable AQE's skew-join handling, which splits oversized partitions at runtime; check for null or default-value keys — a huge share of real-world skew — and filter or handle those rows separately; finally, salt the key: append a random suffix on the skewed side, replicate the other side across the salt range, and aggregate in two phases. Salting is the classic manual answer, but state the trade-off — it multiplies data and complicates the code, so it comes after broadcast and AQE.

A plain Python UDF forces each row out of the JVM: values are pickled, sent to a Python worker process, evaluated row by row, and serialized back. Worse than the serialization cost, the UDF is a black box to Catalyst — it cannot push filters past it, prune the columns it touches, or include it in code generation, so the whole plan degrades. Prefer built-ins from pyspark.sql.functions, which run inside the JVM plan; most string, date, array, and conditional logic is expressible with them, and expr() covers awkward cases. If Python is unavoidable, use a pandas UDF (@pandas_udf): it transfers data via Arrow in columnar batches and evaluates vectorized, typically several times faster — though still opaque to the optimizer.

Write with df.write.partitionBy('event_date').parquet(path), which lays data out as event_date=2026-01-01/ directories. Read with spark.read.parquet(path); a filter on event_date then scans only matching folders — partition pruning — which is the main reason to partition. Details interviewers probe: the partition column lives in directory names, not the files, and is reconstructed with an inferred type on read; pick low-cardinality columns you actually filter on — partitioning by user_id creates millions of directories and the small-files problem; and repartition by the partition column before writing (or set maxRecordsPerFile) so each directory holds a few well-sized files rather than one tiny file per task.

Columnar formats — Parquet most commonly, or ORC — because they enable column pruning (read only the columns the query selects), predicate pushdown (skip row groups whose min/max statistics rule them out), and strong compression, cutting I/O by orders of magnitude versus CSV or JSON. They also carry the schema in the file footer, so reads need no schema inference; inferSchema on CSV costs an extra full pass over the data. Nuances worth naming: pushdown only helps when data is clustered enough for min/max stats to be selective; CSV/JSON are fine as ingestion sources but should be converted early; and Avro is the row-oriented counterpart suited to streaming and Kafka payloads rather than analytical scans.

Catalyst is Spark SQL's query optimizer, and walking its phases is the expected answer: analysis resolves column and table names against the catalog; logical optimization applies rule-based rewrites — predicate pushdown, column pruning, constant folding, join reordering; physical planning converts the logical plan into executable operators, choosing join strategies (broadcast vs sort-merge) partly on cost; finally whole-stage code generation fuses operators into tight JVM bytecode. This is why DataFrames beat hand-written RDD code — Spark rewrites your program. Pre-empt two follow-ups: UDFs are opaque to Catalyst and block these rewrites, and df.explain(True) shows every plan stage. AQE extends Catalyst by re-optimizing at runtime with real statistics.

The driver runs your main program: it owns the SparkSession, turns each action into a DAG of stages, schedules tasks, and receives anything you collect. Executors are JVM worker processes that run the tasks (one per partition per stage) and hold cached blocks and shuffle data. Consequences interviewers look for: collect() and toPandas() pull the whole dataset into the driver — the top cause of driver OOM; code inside transformations runs on executors, so closures are serialized and shipped, which is why unpicklable objects like DB connections must be created inside mapPartitions rather than referenced from driver scope; and broadcast variables exist to ship read-only data to each executor once instead of with every task.

First establish where it dies: a driver OOM points to collect()/toPandas() or an oversized broadcast; an executor OOM points to data-level causes. Then read the Spark UI for the failing stage — task duration and input-size distributions expose skew (one task with 10x the data), and spill metrics show memory pressure. Likely prod-vs-dev culprits: much larger input with the same partition count, so partitions no longer fit; a key skewed only in real data (nulls, defaults); a broadcast that fit dev tables but not prod; explode() multiplying rows. Fix in this order: select needed columns early, repartition to more and smaller partitions, correct or remove bad broadcasts, handle skew via AQE or salting — and only then raise executor memory, which treats the symptom.

AQE re-optimizes the physical plan at runtime using the true statistics available after each shuffle, instead of trusting pre-execution estimates. Its three headline moves: coalescing many small post-shuffle partitions into sensibly sized ones (ending the spark.sql.shuffle.partitions guessing game), switching a sort-merge join to a broadcast join when a side turns out small after filters, and splitting skewed partitions so one hot key stops stalling a stage. It operates at stage boundaries because that is where real sizes become known; it is controlled by spark.sql.adaptive.enabled and is on by default in modern Spark 3 releases. Pre-empt the follow-up: AQE reduces but does not eliminate tuning — a pathological hot key or bad join order can still need manual salting or hints.

Rank rows within each key by recency with a window, then keep the first: w = Window.partitionBy('id').orderBy(col('updated_at').desc()); df.withColumn('rn', row_number().over(w)).filter('rn = 1').drop('rn'). The trap this question really tests: dropDuplicates(['id']) keeps an arbitrary row per key — whichever execution happens to encounter first — so it is nondeterministic and wrong for 'latest record'. Be explicit about ties too: if two rows share the max timestamp, add a tie-breaker column to the orderBy for stable results (or use rank() deliberately if you want all tied rows). The window shuffles once by key, far cheaper than the naive alternative of joining the table back to its own groupBy(max(timestamp)).

groupBy().agg() collapses each group into a single output row — you lose the original rows and keep one aggregate per key. A window aggregation over Window.partitionBy(...) computes the aggregate alongside every original row, which is what running totals, rankings, deltas against a group average, and latest-per-key logic need. Example: avg('salary').over(Window.partitionBy('dept')) attaches the department average to each employee row; the groupBy version yields one row per department and forces a join to get back to row level. Performance notes interviewers probe: both shuffle by the key, but a window with orderBy also sorts within partitions, and a window with no partitionBy at all pulls the entire dataset into a single partition — a classic silent bottleneck.

I work top-down: the Jobs tab to find which job regressed, then its slowest stage, then that stage's task summary metrics. The min/median/max row is usually where the answer is. If max duration is many times the median, it is skew. If every task is slow with high GC time and heavy spill, it is memory. If the stage shows a huge shuffle read, the problem is upstream in the plan. Then the SQL tab for the physical plan: how many files the scan touched, rows in versus out, and whether the join is a BroadcastHashJoin or a SortMergeJoin. Executors tab last, because dead executors usually mean spot reclaim rather than your code. I change config only after the UI has told me which of those it is.

Parquet is columnar, typed and compressed, with min/max statistics per row group, so Spark reads only the columns a query touches and skips row groups whose statistics cannot match the filter. CSV and JSON are row-oriented and untyped: every byte is read, and the schema has to be inferred in an extra full pass unless you supply it. Gzipped CSV is not even splittable, so one file means one core no matter how large the cluster. The practical effect is that column pruning and predicate pushdown stop working entirely on JSON. Parquet's weakness is updates — changing one row means rewriting a file — which is exactly why lakehouse formats like Delta exist on top of it.

Delta is Parquet files plus a transaction log, and that log is what makes the table ACID. On plain Parquet a failed write leaves half-written files that the next reader happily picks up, two concurrent writers silently clobber each other, and deleting one user's rows means finding and rewriting partitions by hand. Delta gives atomic commits, optimistic concurrency control, MERGE, UPDATE and DELETE, schema enforcement, time travel and OPTIMIZE. The cost is real: the log needs its own maintenance, and small files still hurt you. It is worth knowing which side of the market you are interviewing for — Databricks-based GCC teams assume Delta fluency, while plenty of services projects are still plain Parquet on a Hive Metastore.

Time travel reads an older committed version — VERSION AS OF or TIMESTAMP AS OF — because the log stores the exact file list at every commit. VACUUM is what ends it: it deletes data files no longer referenced by the current version and older than the retention threshold, seven days by default. Drop that to zero and you can break in-flight readers and streams mid-query, which is why Delta makes you disable a safety check to do it. The trap people miss is that the log expires separately, thirty days by default, so a very old version can be unreadable even when its files still exist. Day to day I use it for one thing: RESTORE after a bad load, before anyone downstream notices.

OPTIMIZE compacts many small files into large ones; ZORDER BY additionally co-locates rows so the min/max statistics in each file are narrow enough to actually skip files. That second part is the whole point — without clustering, statistics on a high-cardinality column overlap across every file and prune nothing. Z-order on columns you genuinely filter by, two or three at most, since each extra column dilutes the others. It is a full rewrite, so it belongs in a scheduled off-peak job, not in every load. Liquid clustering is the newer answer: CLUSTER BY instead of partitioning plus Z-order, incremental rather than a full rewrite, and you can change the clustering keys later without rewriting history.

Enforcement rejects a write whose schema does not match the table; evolution lets the table absorb the change, but only when you opt in with mergeSchema. The default is enforcement, and that is what you want in silver and gold. If an upstream team renames a column, you want a failed job and a page, not a new column quietly filling with nulls while the old one goes dead. Evolution is appropriate in bronze, where the job is to capture whatever arrived. It only handles additive changes cleanly — an int becoming a string still fails, and dropping a column needs overwriteSchema. Teams that set mergeSchema everywhere end up with amount, amount_new and Amount in the same table, and nobody remembers which one the dashboard reads.

Bronze is raw and append-only, as received; silver is cleaned, typed, conformed and deduplicated; gold is business aggregates. Deduplication belongs in silver. Bronze has to stay a faithful replay log — the day you discover your dedup key was wrong, the only way to fix history is to reprocess from bronze, and you cannot do that if bronze was already deduplicated. In silver I dedup on the business key keeping the latest event timestamp, usually through a MERGE so reruns stay idempotent, or dropDuplicatesWithinWatermark for streaming sources. Gold reads silver only, never bronze. The line I would give an interviewer: bronze is about capture, silver is about truth, gold is about meaning.

A watermark tells Spark how late an event may arrive, which lets it finalise windows and, more importantly, drop state. withWatermark on the event-time column makes Spark track the maximum event time seen and close windows older than that minus the allowed delay; anything later is dropped silently, so if the business cares about late data, count it before it disappears. Without a watermark, windowed aggregations and stream-stream joins hold state forever: the query runs beautifully for two days and then the executors start dying, which is the classic production failure. It has to be event time, not processing time, and the delay you choose is a straight latency-versus-completeness trade you should be able to defend.

The checkpoint holds source offsets, the commit log and the state store; delete it and the query replays from the beginning. But plenty of changes invalidate it without anyone deleting anything. Changing a stateful query's shape breaks resume — adding or removing aggregation keys, changing output mode, swapping or adding a source. For stateful queries the state is partitioned by spark.sql.shuffle.partitions, so changing that number after the query has started needs a fresh checkpoint. Adding a filter or a stateless column is fine. Two rules I hold to: never point two queries at one checkpoint, and never truncate a table without retiring its checkpoint, or the stream believes that data was already processed and you lose it silently.

Structured Streaming is micro-batch with latency in seconds; Flink is a true per-event engine with millisecond latency and far richer state and timer control; Kafka Streams is a library running inside your Java service with no cluster at all, Kafka in and Kafka out. I would pick Spark when the team already runs Spark batch and a few seconds is acceptable — one skill set, one codebase for backfill and live. Flink when the SLA is per-event, or you need complex event-time state machines and true windowing semantics. Kafka Streams when a product team owns the transformation and does not want a platform to operate. In India most data engineering openings assume Spark; Flink shows up mainly in fintech and adtech.

I would land raw files in object storage partitioned by ingest date, load them incrementally into a bronze Delta table, MERGE into silver, and aggregate to gold — and I would split the day into hourly loads rather than one 500GB batch, so a failure costs an hour of reprocessing instead of a day. Sizing comes from the shuffle, not the input: target input partitions in the low hundreds of megabytes and read the spill metrics rather than guessing executor counts. Every load must be idempotent per ingest partition, via replaceWhere or a MERGE key, so a rerun is safe. Then the unglamorous parts that keep it alive: daily OPTIMIZE, file-count and skew monitoring, autoscaling job clusters with spot workers, and a documented backfill path.

First, look at the physical plan and confirm whether it is a BroadcastHashJoin or a SortMergeJoin, because a 10,000-row side should be broadcast. If it is sort-merge, find out why: usually the small side has no statistics, or it is the output of a wide transform so Spark cannot estimate its size. Fix with ANALYZE TABLE or an explicit broadcast hint. Second, check whether those 10,000 rows sit in 10,000 files, because listing can dominate everything else. Third, check task skew on the big side's join key, nulls especially, and filter them before the join. Fourth, look at spill and shuffle read for the stage. Only then touch cluster size. The sneaky one I have hit is a key type mismatch, string joined to int, adding a cast that kills pruning.

If the code did not change, the data or the environment did — I check input volume, key distribution, partition count and cluster config drift before touching a line. The memory model then tells me where it broke. Executor heap splits into a reserved slice, a unified region governed by spark.memory.fraction that execution and storage share (cached blocks get evicted for execution, not the reverse), and user memory for your own objects. Separately there is off-heap overhead, where PySpark worker processes and Arrow buffers live — a Python-side blow-up appears as the container being killed by YARN or Kubernetes, not as a JVM OutOfMemoryError. Usual culprits: one newly skewed key, a bigger array column from upstream, or a cached DataFrame pinning storage.

A job is idempotent when rerunning it for the same input window leaves the table in exactly the same state, which means overwriting a partition or merging on a key — never a bare append. For batch I use replaceWhere or dynamic partition overwrite scoped to the date being processed, or a MERGE on a natural event id. For streaming, checkpoint offsets plus foreachBatch keyed on batchId give the same property. The things that quietly destroy it: current_timestamp() baked into a derived column, monotonically_increasing_id used as a surrogate key since it changes with partitioning, and side writes to a second table from inside the same job. This is the question that separates people who have had to rerun a failed load at 3am from people who have not.

Almost always layout and cluster policy, not clever code. First, stop reading what you do not need: columnar format, partition or cluster on the columns you actually filter by, and select only the columns used — most pipelines scan whole tables to produce a narrow output. Second, kill small files, because listing and per-task overhead can cost more than the compute. Third, move interactive clusters to job clusters with autoscaling and auto-termination; the most expensive line item I have seen was a dev cluster left running over a weekend. Fourth, spot instances for workers, on-demand for the driver. Fifth, make gold incremental — a lot of tables are fully rebuilt every day for no reason at all. Tag jobs so you can prove the before and after.

Speculative execution relaunches a task running far slower than its peers on another executor and keeps whichever finishes first — and that is exactly why accumulator values cannot always be trusted. Spark applies accumulator updates once for tasks inside actions, discarding failed and losing speculative copies, but updates made inside transformations can be applied more than once whenever a stage is recomputed or a task retried. So accumulators are fine for rough diagnostics and useless for a number you report to the business; if the count matters, compute it with an aggregation. Two more things worth saying: speculation does not fix skew, because the duplicate task is just as slow, and it is dangerous when tasks have side effects like calling an external API.

Lineage is the record of which operations produced a DataFrame from its sources; the DAG is the physical graph of stages and tasks that the scheduler builds from that lineage and actually runs. Fault tolerance falls out of the first: when an executor dies, Spark does not need a replica — it recomputes only the lost partitions by replaying lineage from the last available shuffle output, or from the source. That is also why lineage has a cost. Iterative code that keeps appending builds a chain long enough to slow down planning, which is what checkpoint() exists to truncate; cache() does not truncate it, because a cached block can be evicted and the lineage is the fallback. Non-deterministic sources break the guarantee, since recomputation must return the same rows.

Below roughly a hundred gigabytes — and honestly often well above that — a single machine running DuckDB, Polars or even Postgres will beat Spark, because you skip JVM startup, cluster provisioning and network shuffle entirely. Spark earns its overhead when the data genuinely exceeds one machine, when a job runs long enough that you need fault tolerance mid-flight, or when you need the surrounding ecosystem: Delta, streaming, MLlib. It is the wrong tool for low-latency point lookups, which belong in a key-value store or an OLTP database, and for sub-second dashboards, which belong in a serving layer like ClickHouse or Druid. Saying this in an interview reads as judgment rather than a gap — most panels have seen a 2GB job running on a 20-node cluster.