Data Engineering · PySpark

PySpark Production Challenges What breaks, and how to fix it

A Spark job that runs fine on a sample can fail on real data. We debug the failures that actually happen in production, and the questions interviewers ask about them.

A Spark job that runs fine on a sample can fail on real data. In this live session we debug six production failures, each from symptom to fix: data skew (hot keys, salting, skew joins with AQE), slow shuffles and joins (broadcast against sort-merge, partition tuning), out-of-memory crashes (executor sizing, spill and the collect() trap), the small files problem (compaction, coalesce against repartition), reading the Spark UI to find the bottleneck, and finally how to explain all of it in an interview: symptom, root cause, fix, result. Bring your questions: live Q&A at the end. Outside India that is Saturday 17 October at 12:00 AM PT, 2:00 AM CT, 3:00 AM ET, 8:00 AM BST, 9:00 AM CEST, 11:00 AM GST, 3:00 PM SGT and 6:00 PM AEDT.