PART 3 · ADVANCED PYTHON · CHAPTER 9

Iterators and generators: lazy pipelines

FreeLesson 33 of 78

A generator produces values one at a time, only when asked. That single idea lets a laptop process files bigger than its memory.

A THREE-STAGE LAZY PIPELINEPYTHON 3.12

import csv

def read_rows(path):
    with open(path, newline="") as f:
        yield from csv.DictReader(f)       # one row at a time

def delivered(rows):
    for r in rows:
        if r["status"] == "delivered":
            yield r

def amounts(rows):
    for r in rows:
        yield float(r["amount"])

pipeline = amounts(delivered(read_rows("orders.csv")))
print(type(pipeline).__name__)    # nothing has run yet
print(sum(pipeline))
print(sum(pipeline))              # already used up

Output

generator
18972.0
0

Each stage pulls one row from the stage before it. At no point does the whole file sit in memory.

WHO RUNS WHENPYTHON 3.12

def numbers():
    for i in range(3):
        print("  produce", i)
        yield i

for n in numbers():
    print("consume", n)

Output

  produce 0
consume 0
  produce 1
consume 1
  produce 2
consume 2

MEMORY: LIST VS GENERATORPYTHON 3.12

import sys

as_list = [x for x in range(1_000_000)]
as_gen = (x for x in range(1_000_000))
print(sys.getsizeof(as_list), "bytes")
print(sys.getsizeof(as_gen), "bytes")
print(sum(as_gen) == sum(as_list))

Output

8448728 bytes
192 bytes
True

The list holds a million references. The generator object stays the same size whatever range you give it.

NEXT(), STOPITERATION AND YIELD FROMPYTHON 3.12

nums = iter([10, 20])
print(next(nums), next(nums))
print(next(nums, "done"))      # default, no StopIteration

def flatten(items):
    for x in items:
        if isinstance(x, list):
            yield from flatten(x)  # delegate to a sub-generator
        else:
            yield x
print(list(flatten([1, [2, [3, 4]], 5])))

Output

10 20
done
[1, 2, 3, 4, 5]

List / list comprehension	Generator / generator expression
Memory All items at once	One item at a time
Reusable Yes, iterate many times	No, single pass
len(), indexing Yes	No
Use for Small results you reuse, sorting, random access	Large files, streams, pipelines, anything you feed to sum/min/any

Practice before you move on

Write a generator that yields only the delivered rows of orders.csv, and sum their amounts.
Chain a second generator that turns each amount into a float, and check that nothing runs until you call sum.