Cohort 4Register

PART 2 · PYTHON FOUNDATIONS · CHAPTER 8

Files: CSV, JSON, JSON Lines, paths

FreeLesson 31 of 78

Most DE Python starts with reading a file someone else wrote. Know the standard library before reaching for pandas.

PATHLIB + CSVPYTHON 3.12
from pathlib import Path
import csv

data = Path(".")
print(sorted(p.name for p in data.glob("*.csv"))[:4])
p = Path("orders.csv")
print(p.suffix, p.stat().st_size, "bytes")

with open("orders.csv", newline="", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    print(reader.fieldnames)
    rows = [{**r, "amount": float(r["amount"])} for r in reader]

out = Path("delivered.csv")
with out.open("w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=["order_id", "amount"],
                       extrasaction="ignore")
    w.writeheader()
    w.writerows(r for r in rows if r["status"] == "delivered")
print(out.read_text().splitlines()[:3])

Output

['employees.csv', 'logins.csv', 'order_items.csv', 'orders.csv']
.csv 566 bytes
['order_id', 'customer_id', 'order_date', 'status', 'amount']
['order_id,amount', '101,1198.0', '102,796.0']
JSON VS JSON LINESPYTHON 3.12
import json

with open("events.jsonl", encoding="utf-8") as f:
    for i, line in enumerate(f):          # one record at a time
        event = json.loads(line)
        if i < 2:
            print(event["event"], event["props"]["device"])

customers = json.load(open("customers.json"))   # whole file
print(len(customers), customers[1]["address"])
print(json.dumps({"id": 1, "city": None}))

Output

view mobile
cart mobile
8 {'city': 'Mumbai', 'pincode': '400001'}
{"id": 1, "city": null}
Read it withWhen
for line in fBig text files. Reads one line at a time, constant memory.
f.read() / f.readlines()Small files only. Loads everything into memory.
csv.DictReaderCSV with a header. Handles quoting and embedded commas that split(",") gets wrong.
json.load / json.loadsA file object / a string. Watch the s.
pd.read_csv(..., chunksize=n)Large CSV into pandas in batches (page 70).

Practice before you move on

  1. Convert orders.csv into a JSON Lines file, one order per line, then read it back and count the lines.
  2. Use pathlib to list every .csv file in the folder with its size in bytes.