Cohort 4Register

Free, every lesson

Python for Data Engineering

End to end notes, basic to advanced

From installing Python to DSA, Big-O and the coding patterns DE interviews keep reusing, with a 10-week plan to get there. Every topic comes with the approach, code that runs, and the real output.

Read every lesson here. Run any snippet in your browser on the book's dataset, and practise the 18 patterns in Coding Practice.

By Durgesh Yadav · October 2026 edition

Start readingSee the 18 patterns
stream_orders.pyPYTHON 3.12
def read_orders(path):
    with open(path) as f:
        next(f)                  # skip header
        for line in f:           # 1 line in RAM
            *_, s, a = line.rstrip().split(",")
            yield s, float(a)

delivered = (a for s, a in read_orders("orders.csv")
             if s == "delivered")
print(f"revenue: {sum(delivered):,.2f}")

Output

revenue: 18,972.00

A generator reads one line at a time, so this same code works on a 15-row file and a 15 GB one. See why

lessons, one for each page of the book
78lessons, one for each page of the book
parts, from your first program to the coding round
7parts, from your first program to the coding round
coding round patterns, each with a practice question
18coding round patterns, each with a practice question
solved questions, each with its real output
47solved questions, each with its real output

How a lesson works

Read it, run it, then do it yourself

The book's text

Each lesson is one page of the book, in the book's own words.

Code with its real output

Every output panel is what Python 3.12 printed for that code on the book's dataset.

Try it in your browser

Open a snippet in an editor on the page and run it. The dataset's files are in its folder, so open("orders.csv") works.

Exercises and a video

Two or three exercises close each lesson, with the matching free video lesson where there is one.

Contents

78 lessons in the book's order

Never written Python? Start at the top. Already code in Python? Skim Part 1 and start at Part 2.

Before you start

Start here

the running dataset, and what the coding round asks

  1. 1Start here: the running dataset
  2. 2The reality: what DE coding rounds ask

Part 1

Python from zero

install, variables, loops, lists, dicts, functions

  1. 3What Python is, and running your first program
  2. 4Variables and names
  3. 5Numbers and maths
  4. 6Strings: working with text
  5. 7True, False and comparisons
  6. 8Input and output
  7. 9Making decisions: if, elif, else
  8. 10Loops part 1: for and range
  9. 11Loops part 2: while, break, continue
  10. 12Lists: ordered collections
  11. 13Tuples and sets
  12. 14Dictionaries: key and value
  13. 15Which data structure? A side-by-side guide
  14. 16Functions: reusable steps
  15. 17Functions part 2: lambdas, sorting and building a pipeline
  16. 18Modules, packages and pip
  17. 19Reading errors without panic
  18. 20Files: read and write
  19. 21Mini project: a monthly sales report
  20. 22Practice set 0: first steps

Part 2

Python foundations

objects, dicts, files, errors

  1. 23The 6-step approach to any coding question
  2. 24Names, objects and mutability
  3. 25Numbers, strings and dates that bite
  4. 26Lists, tuples, sets and Big-O
  5. 27Dicts, defaultdict and Counter
  6. 28Loops, comprehensions and unpacking
  7. 29Functions, scope and closures
  8. 30Errors, exceptions and context managers
  9. 31Files: CSV, JSON, JSON Lines, paths
  10. 32Practice set 1: foundations

Part 3

Advanced Python

generators, OOP, async

  1. 33Iterators and generators: lazy pipelines
  2. 34itertools, functools and sorting
  3. 35Classes and dataclasses for pipelines
  4. 36Decorators: timing, retry, context managers
  5. 37Concurrency: threads, processes, asyncio
  6. 38Practice set 2: advanced

Part 4

DSA and complexity

Big-O, hashing, trees, graphs, DP

  1. 39Big-O in plain words: how work grows
  2. 40Space complexity: memory, recursion, streaming
  3. 41Time vs space trade-offs
  4. 42What Python operations really cost
  5. 43Arrays and strings: two pointers, prefix sums
  6. 44Hashing: two-sum, anagrams, subarray sums
  7. 45Sliding windows of variable size
  8. 46Stacks and queues: brackets, monotonic stacks
  9. 47Linked lists: the three classics
  10. 48Binary search, including search on the answer
  11. 49Sorting: what Python does, and when to sort yourself
  12. 50Recursion and backtracking
  13. 51Trees: DFS, BFS, and the org chart
  14. 52Graphs: BFS, cycles and topological sort
  15. 53Heaps and greedy choices
  16. 54Dynamic programming: four classics
  17. 55Practice set 3: DSA with complexity

Part 5

Coding round patterns

18 patterns, solved

  1. 56The pattern map: 18 patterns at a glance
  2. 57P1 Frequency count · P2 Top-K
  3. 58P3 Group by key · P4 Dedupe, keep latest
  4. 59P5 Sliding window · P6 Merge sorted streams
  5. 60P7 Flatten nested JSON
  6. 61P8 Sessionize events · P9 Consecutive streaks
  7. 62P10 Parse logs with regex
  8. 63P11 Merge intervals · P12 Hash join
  9. 64P13 Chunked processing · P14 Retry with backoff
  10. 65P15 Paginated extraction · P16 Memoize, LRU cache
  11. 66P17 Validate and quarantine · P18 Diff two snapshots

Part 6

Python for data engineering

pandas, SQL, ETL, PySpark

  1. 67pandas essentials, mapped to SQL
  2. 68pandas pitfalls and performance
  3. 69Python and databases: parameters, transactions, upserts
  4. 70Build an ETL pipeline: idempotent, logged, configurable
  5. 71Testing with pytest and data quality checks
  6. 72From pandas to PySpark, scheduled with Airflow

Part 7

Practice and prep flow

10-week plan, cheat sheets

  1. 73Interview set: Python coding for DE
  2. 74Interview set: pandas and data wrangling
  3. 75Concept questions with short answers
  4. 76Complexity cheat sheet
  5. 77The preparation flow: 10 weeks, zero to offer
  6. 78Revision sheet and the last 48 hours

Part 5

The 18 coding round patterns

Each one is solved in its lesson, with how the question sounds and how to attack it. Each also has a practice question in Coding Practice, under Python, graded on hidden tests.

  1. P1Frequency count

    Sounds like: "how many of each status", "most common error message", "count events per page"

    Practice: Most visited pages from an event stream

  2. P2Top-K

    Sounds like: "top 3 customers by spend", "10 largest files", "k most frequent words"

    Practice: Top K customers by delivered spend

  3. P3Group by key

    Sounds like: "total per order", "list of items per user", "reconcile header vs lines"

    Practice: Check basket totals against order amounts

  4. P4Dedupe, keep latest

    Sounds like: "remove duplicates", "latest version of each record", "the CDC feed has repeats"

    Practice: Keep the latest version of each order

  5. P5Sliding window

    Sounds like: "rolling 3-day revenue", "moving average of latency", "max sum of k consecutive"

    Practice: Rolling 3-day revenue with a sliding window

  6. P6Merge sorted streams

    Sounds like: "merge two sorted log files", "combine web and app events in time order"

    Practice: Merge sorted event streams

  7. P7Flatten nested JSON

    Sounds like: "flatten this API payload", "turn nested JSON into table rows", "explode the items array"

    Practice: Flatten a nested order and explode its items

  8. P8Sessionize events

    Sounds like: "split clickstream into sessions", "a session ends after 30 minutes idle", "average session length"

    Practice: Sessionize click events with a 30-minute gap

  9. P9Consecutive streaks

    Sounds like: "longest login streak", "most consecutive days with sales", "longest consecutive sequence"

    Practice: Longest run of consecutive login days

  10. P10Parse logs with regex

    Sounds like: "from this log, report errors per job", "which step is slowest", "parse these lines into records"

    Practice: Parse pipeline logs into a per-job report

  11. P11Merge intervals

    Sounds like: "merge overlapping bookings", "find busy periods", "total time covered by sessions"

    Practice: Merge room bookings into occupied periods

  12. P12Hash join

    Sounds like: "join these two files without pandas", "enrich orders with customer data"

    Practice: Hash join orders to customers in plain Python

  13. P13Chunked processing

    Sounds like: "the file is 50 GB and memory is 8 GB", "insert in batches of 1,000", "process an S3 object stream"

    Practice: Process rows in chunks with a running total

  14. P14Retry with backoff

    Sounds like: "the API returns 429 sometimes", "the warehouse times out at night", "make this call resilient"

    Practice: Retry a rate-limited API with exponential backoff

  15. P15Paginated extraction

    Sounds like: "pull every page from this API", "the endpoint returns 100 rows and a next link"

    Practice: Extract every row from a cursor-paginated API

  16. P16Memoize, LRU cache

    Sounds like: "avoid repeat API lookups", "cache dimension lookups", "design an LRU cache"

    Practice: Build an LRU cache from scratch

  17. P17Validate and quarantine

    Sounds like: "reject bad rows but load the rest", "enforce a data contract", "where do bad records go?"

    Practice: Validate rows and quarantine the bad ones

  18. P18Diff two snapshots

    Sounds like: "what changed since yesterday", "build a CDC feed from daily dumps", "feed an SCD Type 2 load"

    Practice: Diff two daily snapshots into inserts, deletes and updates

Worked examples

47 solved questions

Each one with the approach, the code and what it printed. Try one before you open its lesson.

Practice sets

  1. Q1Print the numbers 1 to 15, but "Fizz" for multiples of 3, "Buzz" for multiples of 5 and "FizzBuzz" for both.
  2. Q2Find the largest and smallest amount without using max() or min().
  3. Q3Count how many times each word appears in a sentence, ignoring case.
  4. Q4Check whether a string reads the same backwards, ignoring spaces and case.
  5. Q5Given a list of order statuses, return each status with the share of orders it makes up, as a percentage.
  6. Q6From orders.csv, return each customer's total delivered spend, highest first. Ties broken by customer_id.
  7. Q7Total amount and order count per status from orders.csv, without pandas.
  8. Q8Which customers in customers.json have never placed an order?
  9. Q9Count events per device from events.jsonl. The device sits inside a nested props object.
  10. Q10Clean these amount strings into Decimal, skipping the ones that cannot be parsed: "₹1,198.00", " 796 ", "N/A",
  11. Q11From employees.csv, find the second-highest distinct salary. Return None if it does not exist.
  12. Q12Write chunked(iterable, n) that yields lists of n items, without itertools.batched. It must work on a generator.
  13. Q13Run-length encode a status stream: ["D","D","C","D","D","D"] becomes "D2C1D3".
  14. Q14Write a decorator count_calls that records how many times each decorated function ran.
  15. Q15Sort employees.csv by department A to Z, then salary high to low, then name. Print the top two per
  16. Q16Order ids should be 1 to n with exactly one missing. Find it.
  17. Q17Find the k-th largest order amount without sorting the whole list.
  18. Q18Return the longest common prefix of these S3 paths.
  19. Q19For each day, the product of every other day's growth factor, without division.
  20. Q20Find files in a folder that have exactly the same content, even under different names.

Worked questions of the patterns

  1. P1Count orders per status and per (customer, status), then the two most visited pages from the event stream.
  2. P2The top 3 customers by delivered spend, first with the library, then with a heap you write yourself.
  3. P3Group order_items by order, total each basket, and check every total against orders.amount.
  4. P4A change feed repeats orders, and one older record arrives after a newer one. Keep the latest version per order.
  5. P5A rolling 3-day revenue total over daily delivered revenue for 1 to 7 April (zeros already filled in).
  6. P6Merge web and app event streams, each already sorted by time, into one ordered stream. Then write the merge by hand.
  7. P7Flatten customer records with a nested address and a tags list into flat columns, then explode an order's items into one row each.
  8. P8Group each user's events into sessions, where a gap of more than 30 minutes starts a new session.
  9. P9Each user's longest run of consecutive login days.
  10. P10Parse app.log into records, skip anything malformed, then report per job: lines, errors, slowest step and the most common error.
  11. P11Merge each room's bookings into continuous occupied periods, then solve the classic numeric version.
  12. P12Join orders to customers in plain Python, count orphan orders, and list customers who never ordered.
  13. P13Process orders.csv four rows at a time (pretend it is 50 GB) and keep a running delivered total.
  14. P14Retry a rate-limited API with exponential backoff and full jitter, and give up after a fixed number of tries.
  15. P15Extract every record from a cursor-paginated API (simulated, so it runs here) as one stream of rows.
  16. P16Cache pincode lookups so repeats cost nothing, then build an LRU cache from scratch.
  17. P17Check every row of orders_dirty.csv against the rules, load the good ones, and park the bad ones with reasons.
  18. P18Compare two daily customer snapshots and emit inserts, deletes and field-level updates.

Interview sets

  1. Q39From events.jsonl, return each user's latest event and the device it came from.
  2. Q40For each customer, the longest gap in days between two consecutive orders.
  3. Q41Find every pair of orders whose amounts add up to exactly 2847.00.
  4. Q42Unflatten {"address.city": "Pune", "address.pincode": "411001", "name": "Rohan"} back into nested JSON.
  5. Q43The top customer by delivered spend in each month.
  6. Q44Month-over-month growth in delivered revenue, as a percentage.
  7. Q45Daily delivered revenue for 1 to 7 April with the missing days as 0, plus a 3-day rolling sum.
  8. Q46Customers whose every order was delivered.
  9. Q47For each customer: first order date, number of orders, and days until their second order.

The dataset

One small store, from the first lesson to the last

Every snippet reads these files by name, for example open("orders.csv"). Try it puts them in the folder Python runs in.