44.9%
Revenue overstated by one join fan-out. The query has no error and passes review.
Chapter 3
New · Cohort 4AI-Powered Data Engineering Cohort 4 goes live 26 September · only 40 seats · Sat–Sun · 7 AM–10 AM ISTRegister Now
Free book · 244 pages · PDF · No card, no trial
Data EngineeringFrom Zero to Production & Interview Ready
The pipeline ran successfully.The numbers were wrong.
Most people meet data engineering backwards: a tool first, then another, and six months later they still cannot say what happens between a customer tapping Place Order and that order appearing on a dashboard. This book runs the other way round. It follows one order through a fictional Indian e-commerce company and lets each tool arrive at the moment the story needs it.
By Durgesh Yadav · Senior Data Engineer · Educator · Mentor · Founder, PrepNPlaced · First edition · 2026
- One Google sign-in
- Never uses your free note in the vault
- Field Kit included
- pages, cover to index
- 244
- pages, cover to index
- chapters — Lecture Zero plus fourteen
- 15
- chapters — Lecture Zero plus fourteen
- parts, from the terminal to the interview loop
- 6
- parts, from the terminal to the interview loop
- of the figures measured, not asserted
- 100%
- of the figures measured, not asserted
Measured, not asserted
Four numbers, each produced by code that was actually run
Almost every figure in the book came from a script executed while it was being written. The scripts are in the companion Field Kit, so you can reproduce them, change them, and check the book.
34.8×
Smaller on disk, CSV to Parquet with zstd, on the same two million rows.
Chapter 6
₹4,749
Moved between two cities, retroactively, by one slowly-changing-dimension decision.
Chapter 5
3×
Revenue tripled by a load that ran three times. Every run reported success.
Chapter 8
Why it runs the other way round
A tool you meet as the answer to a problem you already have is a tool you remember
One order, followed end to end
Kafka appears when the order database starts buckling under six teams querying it. Spark appears when a Python script that took ten minutes starts taking nine hours. Airflow appears when there are forty scripts and nobody remembers which runs first.
Simpler explanations, not simpler engineering
Being a beginner means you need clearer explanations, not a shallower field. When the book reaches Spark it does not stop at “it’s fast because memory”. It reads the real execution plan, finds the word Exchange, and explains why one slow task holds up a job that has finished 199 of its 200 pieces.
The failure that defines the job
A pipeline that crashes is a good pipeline having a bad day. The one that succeeds while producing wrong numbers is the failure this profession is organised around, and every chapter is built to make you see it coming.
Who this book is for
It assumes almost nothing. It refuses to be shallow.
If you have finished Class 12 and can use a computer, you can read this. You do not need a computer science degree.
The non-CS graduate or Class 12 student
Start at Lecture Zero and skip nothing. The foundations are built from scratch, in the specific way a data engineer needs them.
The beginner programmer
Skim Lecture Zero, then slow down at Chapter 3. SQL is the single highest-return skill here.
The analyst moving into engineering
Your SQL is fine. Your gap is pipelines, orchestration, distributed processing, and guaranteeing tables instead of consuming them.
The software engineer moving across
Chapter 2 is revision. Data modeling and silent failure are the parts software engineers most often get wrong.
The data engineer preparing for interviews
Chapter 14 has the loop, but the reasoning lives in the chapters the questions come from.
The working engineer patching fundamentals
The "what happens behind the scenes" sections and the measured results are written for you.
What is inside
15 chapters in six parts
Read it in order, at least once. The book builds one continuous example and one continuous vocabulary: Chapter 9 assumes what a partition meant in Chapter 6, which assumed what a file format meant in Chapter 4.
Part 1
Foundations
Before any tool: the terminal you will live in, the job you are actually applying for, and the four physical resources every slow pipeline is waiting on.
- 0
Lecture Zero
Living in the Terminal
About thirty commands, the exit code every scheduler reads, and the habit of looking at a file before you load it.
15 sectionsp. 14 - 1
Chapter One
What a Data Engineer Actually Does
The question every interviewer asks, OLTP against OLAP at the level of bytes, and the failure mode that defines the discipline.
11 sectionsp. 26 - 2
Chapter Two
The Machine Underneath
CPU, memory, disk, network. Decide whether a slow job is busy or waiting before you change anything.
12 sectionsp. 41
Part 2
The Two Languages
SQL and Python are the whole toolkit for the first year. The level that gets you hired sits well past where most tutorials stop.
- 3
Chapter Three
SQL to Interview Level
The join that overstates revenue by 44.9% with no error, the NULL trap that eats candidates, window functions, and the eight patterns interviewers actually ask.
12 sectionsp. 54 - 4
Chapter Four
Python for Data Engineers
A seventy-line ingestion script that quarantines bad rows, logs its counts, closes what it opens, and is safe to run twice.
13 sectionsp. 76
Part 3
Shaping Data
What one row means, where the bytes physically sit, and whose machines they sit on.
- 5
Chapter Five
Data Modeling
Grain, star schemas, surrogate keys, and the slowly-changing-dimension decision that moved ₹4,749 between two cities overnight.
13 sectionsp. 93 - 6
Chapter Six
Where Data Lives
CSV to Parquet, measured: 34.8× smaller and 10× faster. Partitioning, the small files problem, and what Delta and Iceberg actually add.
12 sectionsp. 109 - 7
Chapter Seven
The Cloud, and What You Are Actually Renting
The six things you rent, a translation table across AWS, Azure and GCP, and a cost model you can do in your head.
14 sectionsp. 121
Part 4
Moving Data
Loading the same file twice, processing more than one machine can hold, and running forty jobs that depend on each other.
- 8
Chapter Eight
Pipelines: ETL, ELT and Loading Data Twice
Watermarks, idempotency, MERGE done properly, change data capture, late data, dbt, and the load that tripled revenue in three runs.
12 sectionsp. 137 - 9
Chapter Nine
Apache Spark
Read a physical plan, find the word Exchange, broadcast the small side, and fix skew with AQE and salting.
13 sectionsp. 153 - 10
Chapter Ten
Orchestration with Airflow
DAGs, logical dates, sensors that deadlock, and a real run where the quality gate stopped a bad table from publishing.
10 sectionsp. 166
Part 5
Running It
Streams that never stop, and the failure that defines this profession: the pipeline that succeeds while producing wrong data.
- 11
Chapter Eleven
Streaming and Kafka
Partitions and keys, consumer lag, delivery semantics, event time against processing time, and when not to stream at all.
9 sectionsp. 177 - 12
Chapter Twelve
When Pipelines Lie
Monitor the data, not the job. Six checks, data contracts, the debugging framework, and production as a practice rather than a place.
10 sectionsp. 187
Part 6
Getting Hired
One project you built and broke on purpose, and the ability to reason out loud under mild pressure.
- 13
Chapter Thirteen
The Portfolio Project
Five stages from an empty folder, then broken on purpose four ways, with the actual output of every run.
14 sectionsp. 200 - 14
Chapter Fourteen
The Interview Loop
A system-design framework, three designs worked end to end, the loop round by round, and thirty days of revision.
6 sectionsp. 221
Plus a glossary, the Field Kit, further reading, and an index.
Every chapter ends with the same boxes: common beginner mistakes, in the interview answered in four layers, and practice tasks with no printed solutions. That last part is deliberate.
Look inside
Four pages, exactly as they print
Real query results, real execution plans, real break tests. Not a diagram of what should happen.
Page 63
The fan-out: 44,036 against 30,390, and the rule that prevents it forever.
Page 103
Two correct dashboards, two different answers. Which one finance should read.
Page 158
A real physical plan, and the one word to look for in every plan you read.
Page 211
The portfolio project, broken on purpose. What actually happened.
The Field Kit
Every number in the book was produced by code that was actually run. That code is yours.
Run it, change the numbers, and check the book. The kit is organised by chapter so a claim on page 63 is one folder away from the script that made it.
The verification scripts
The ShopKart dataset and every script behind every measured result: the 44.9% join fan-out, the SCD Type 2 comparison, the storage benchmarks, the load that tripled revenue, the Spark execution plans, and the Kafka partitioner.
The portfolio project
The mock API that rate-limits and sends malformed rows on purpose, all five stages from Chapter 13, the Airflow DAG, and the four break tests — so the project can be built offline with nothing but Python and DuckDB.
The interview question bank
Data engineering and data analytics questions organised by topic, with answers, on PrepNPlaced — the same bank that powers the interview-prep tools.
Open the question bankOpen source, MIT licensed, one folder per chapter.
Clone it, run the scripts, and open an issue if a number does not reproduce.
About the author
Durgesh Yadav
Senior Data Engineer · Educator · Mentor · Founder, PrepNPlaced
Durgesh teaches data engineering and data analytics, and mentors engineers moving into the field — many of them from non-technical backgrounds, which is the reader this book was written for. The book is the map. The AI-Powered Data Engineering cohort is the guided trek, with live weekend classes and projects reviewed by working data engineers.
Ch 3
SQL to interview level — the chapter most readers are hired on
Ch 12
When pipelines lie — the chapter that changes how interviewers see you
Ch 13
A portfolio project you build, then break on purpose
FAQ
Questions people ask before they download
Is the book really free?
Yes. The full 244-page PDF is free. Downloading takes one Google sign-in through the PrepNPlaced notes vault so we can send you the Field Kit when it updates. The book does not use up your free note in the vault, and there is no card and no trial.
Who is it written for?
Anyone from a Class 12 student or non-CS graduate through to a working engineer patching fundamentals. It assumes almost nothing: Lecture Zero and Chapters 2 to 4 build the terminal, SQL and Python foundations from scratch, in the specific way a data engineer needs them.
How long does it take to work through?
Two to four months at about ten hours a week if you type the code and do the practice tasks. The thirty-day plan in Chapter 14 is a revision schedule for someone who has already read the book, not a first pass.
What is the Field Kit?
The scripts behind every measured figure in the book (the 44.9% fan-out, the 34.8× Parquet reduction, the tripling load, the Spark plans), the ShopKart dataset, the complete Chapter 13 portfolio project with its mock API and break tests, and the Airflow DAG. Published on GitHub so you can reproduce and challenge every number.
Do I need to pay for any tools to follow along?
No. Everything runs on a laptop with Python, DuckDB and a local Spark. The cloud chapter's exercises are sized for the free tiers and start by setting a budget alert.
Is this the same as the AI-Powered Data Engineering cohort?
The book is the map; the cohort is the guided trek. The cohort adds 192 hours of live weekend classes, three GitHub projects reviewed by working data engineers, and interview preparation. Read the book first, then decide.
Is there a print edition?
Not yet. The PDF is the current edition. If a paperback follows, it will be announced on this page first.
You will get more from this book if you argue with it.
When you read that a wide transformation forces a shuffle, do not underline it. Open a terminal, run something that shuffles, and check whether the picture in your head matches what the engine actually did. The gap between those two things is where an engineer is made.