PySpark Projects for Beginners: Six Builds, the Dataset for Each, and the Interview Question It Prepares

Which PySpark projects should you build, and in what order?

Build them in the order interview questions get harder: aggregations, window functions, file formats and partitioning, a layered pipeline, nested data, then streaming. Spark is worth the effort. In PrepNPlaced's India Tech Hiring Report, Spark was named in 17.2% of 1,470 data-engineer postings and Databricks in 19.1% of 1,470, collected 23 August to 27 September 2026. The report counted whole words, so a posting that wrote only "PySpark" is not in the Spark figure.

How do you start Projects 1 and 2 without installing Spark?

Start on PrepNPlaced's Coding Practice, where the PySpark questions run on two datasets: Chinook, a music store's invoices and tracks, and Northwind, a trading company's orders and products. Each table is preloaded as a DataFrame named after it, and your answer goes in result. Run executes your code on a live Spark sandbox and shows the first 50 rows; Submit grades the full output. Running is free; you sign in to submit. For Project 1, solve join and aggregate questions such as "Total revenue by genre". For Project 2, move to window functions on Northwind, such as "Top 3 spending customers within each country". The example below reads the CSV files first, so it also runs outside Coding Practice.

Project 3: how do you clean and partition a real public dataset?

Use the NYC Taxi and Limousine Commission trip records, published monthly as Parquet files with a data dictionary and a taxi zone lookup table in CSV. Read one month of yellow taxi trips and count the rows that make no sense: zero distances, zero amounts, drop-offs before pick-ups. Filter them out, join the zone lookup for borough names, and write the result partitioned by pick-up date. In Databricks, the samples catalog has samples.nyctaxi.trips, though Databricks says sample datasets can change without notice. This project teaches cost. Every groupBy and join shuffles into spark.sql.shuffle.partitions partitions, 200 by default per the Spark tuning guide, and adaptive query execution, on by default since Spark 3.2.0, uses run-time statistics to pick a better plan.

Project 4: what does the PrepNPlaced cricket project add?

It adds the shape most data engineering interviews expect: a layered pipeline on Databricks. PrepNPlaced's PySpark on Cricket Data project takes raw IPL ball-by-ball data into Databricks and models it Bronze, Silver and Gold through a modular PySpark pipeline, tuned with cache(), partitions and shuffle settings, with outputs written to Delta. The paid module, opened by Learn Pass or by buying that session's recording, holds the recording, the class dataset, four notebooks (00 Config, 01 Bronze, 02 Silver, 03 Gold) and the complete notes, whose first pages are readable without signing in. To build it from public files instead, Cricsheet publishes IPL match data as JSON, mostly under the Open Data Commons Attribution License, so credit Cricsheet in your README.

Projects 5 and 6: how do you move on to nested JSON and streaming?

Use GH Archive, which records the public GitHub timeline as hourly gzip-compressed JSON files at data.gharchive.org, named like 2015-01-01-15.json.gz. Spark reads them directly with spark.read.json. Project 5 selects the nested repo fields, drops duplicate events by id, and counts events per repository for one day. Project 6 copies those files into a landing folder one at a time and reads it with spark.readStream. Add a watermark on event time, count events in five-minute windows and write to Delta with a checkpoint. The Spark streaming guide states the trade: late data within the watermark's threshold is aggregated, and later data can be dropped. Restart the query and show that the counts did not double.

How do you put a PySpark project on your resume and GitHub?

Give each project its own repository with a README an interviewer can follow alone, and one resume line naming the data, the stack and a number you measured.

1. 1Name the dataset in the README's first line and link its source.
2. 2Draw the flow in one line, such as raw Parquet to cleaned table to daily summary.
3. 3List the steps that rebuild the output from an empty folder.
4. 4State one check, such as a row count, and what it caught.
5. 5Write the resume line as built X with Y and measured Z.
6. 6Rehearse the data flow, what broke, and what changes at much larger volume.