PART 2 · PYTHON FOUNDATIONS · CHAPTER 8
Files: CSV, JSON, JSON Lines, paths
FreeLesson 31 of 78
Most DE Python starts with reading a file someone else wrote. Know the standard library before reaching for pandas.
from pathlib import Path
import csv
data = Path(".")
print(sorted(p.name for p in data.glob("*.csv"))[:4])
p = Path("orders.csv")
print(p.suffix, p.stat().st_size, "bytes")
with open("orders.csv", newline="", encoding="utf-8") as f:
reader = csv.DictReader(f)
print(reader.fieldnames)
rows = [{**r, "amount": float(r["amount"])} for r in reader]
out = Path("delivered.csv")
with out.open("w", newline="", encoding="utf-8") as f:
w = csv.DictWriter(f, fieldnames=["order_id", "amount"],
extrasaction="ignore")
w.writeheader()
w.writerows(r for r in rows if r["status"] == "delivered")
print(out.read_text().splitlines()[:3])Output
['employees.csv', 'logins.csv', 'order_items.csv', 'orders.csv'] .csv 566 bytes ['order_id', 'customer_id', 'order_date', 'status', 'amount'] ['order_id,amount', '101,1198.0', '102,796.0']
import json
with open("events.jsonl", encoding="utf-8") as f:
for i, line in enumerate(f): # one record at a time
event = json.loads(line)
if i < 2:
print(event["event"], event["props"]["device"])
customers = json.load(open("customers.json")) # whole file
print(len(customers), customers[1]["address"])
print(json.dumps({"id": 1, "city": None}))Output
view mobile
cart mobile
8 {'city': 'Mumbai', 'pincode': '400001'}
{"id": 1, "city": null}| Read it with | When |
|---|---|
for line in f | Big text files. Reads one line at a time, constant memory. |
f.read() / f.readlines() | Small files only. Loads everything into memory. |
csv.DictReader | CSV with a header. Handles quoting and embedded commas that split(",") gets wrong. |
json.load / json.loads | A file object / a string. Watch the s. |
pd.read_csv(..., chunksize=n) | Large CSV into pandas in batches (page 70). |
Practice before you move on
- Convert orders.csv into a JSON Lines file, one order per line, then read it back and count the lines.
- Use pathlib to list every .csv file in the folder with its size in bytes.