START HERE

Start here: the running dataset

FreeLesson 1 of 78

The same online store from the SQL End to End Notes, now as the files a data engineer actually receives: CSV, JSON, JSON Lines and a log file.

These notes take you from Python basics to the patterns that decide DE coding rounds, then into pandas, databases, testing and PySpark. The pattern section comes from the questions I keep seeing across the 20+ analytics and data engineering interviews I have sat, and the ones my mentees report back from their own loops. Every snippet reads the files below, and every output panel is what Python 3.12 printed when this PDF was built. The files ship with this PDF as

python_de_dataset.zip.

The files

File	Format	Records	What's inside	Used for
`orders.csv`	CSV	15	order_id, customer_id, order_date, status, amount	Most of the book
`order_items.csv`	CSV	24	order_id, product_id, qty, unit_price	Group by, joins
`products.csv`	CSV	8	product_id, product_name, category, price	Lookups, pandas
`customers.json`	JSON	8	Nested address {city, pincode}, tags list	Flattening (P7), joins
`events.jsonl`	JSON Lines	12	user_id, ts, event, props {page, device}	Sessions, funnels
`logins.csv`	CSV	12	user_id, login_date	Streaks (P9)
`employees.csv`	CSV	9	emp_id, name, dept, manager_id, salary	Top-K, ties
`app.log`	Text	14	Pipeline log lines, one malformed	Regex parsing (P10)
`orders_dirty.csv`	CSV	7	Bad rows on purpose: negative, blank, bad date	Validation (P17)
`snapshot_day1/2.csv`	CSV	8 / 8	Customers on two days	Change capture (P18)

A first look

A FIRST LOOKPYTHON 3.12

import csv, json

with open("orders.csv", newline="") as f:
    first = next(csv.DictReader(f))
print(first)

customers = json.load(open("customers.json"))
print(customers[5])

Output

{'order_id': '101', 'customer_id': '1', 'order_date': '2025-01-10', 'status': 'delivered', 'amount': '1198.00'}
{'customer_id': 6, 'name': 'Sana', 'signup_date': '2025-03-15', 'referred_by': None, 'address': {'city': None, 'pincode': None}, 'tags': []}

DA data wrangling round DE data engineering coding round EASY MEDIUM HARD difficulty

Practice before you move on

Read orders.csv with csv.DictReader and print the number of rows and the column names.
Print the distinct statuses in orders.csv, then the earliest and the latest order_date.
Read events.jsonl one line at a time and print the set of event types it holds.