How do generator pipelines stream data lazily? | Interview Question

Q: How do generator pipelines stream data lazily?

**Target Audience:** Developers who know basic generators and want to process big or endless data streams; candidates preparing for "how would you process a huge file?" questions. **Difficulty:** Medium **Python version:** Behaviour checked on CPython 3.11.17 and in the page runner (Pyodide 0.25.1, CPython 3.11.3). > **How to read this doc:** Each idea is explained in plain words first, then tagged with `📌 Interview term:` — that is the exact vocabulary an interviewer expects to hear. Every output shown was printed by a real run; the **Verified** section quotes it word for word. ## 1. Why This Even Matters — A Story First Think of a sushi conveyor belt versus a buffet. At a buffet, the kitchen cooks *everything* first and lays it all out; you need a huge table and lots of food sits around. On a conveyor belt, the chef makes a plate only when there is room on the belt, each plate passes one station after another (rice, fish, sauce), and when you say "I am full", the chef simply stop

The problem

Question presented to candidate: "Say you need to process a 50 GB log file: read lines, keep the errors, parse them, and count by type. How would you do it with generators, and why does that not run out of memory?"

What a strong answer should cover:

A pipeline is a chain of generators, each consuming the previous one: count(parse(errors(read(f)))). Building the chain runs nothing; the final consumer pulls one item at a time through every stage.
Memory is constant in the input size — only the current item (plus any per-stage state) is alive — versus a list-based version that materialises every intermediate stage.
Laziness enables infinite sources and early termination: islice, any, next and break stop pulling, so upstream work that is not needed never happens.
Trade-offs: generators are one-shot, errors surface when an item is pulled (not when the pipeline is built), cleanup needs close()/with, and per-item Python call overhead makes them slower than a vectorised or list approach for small data.
Tools: generator functions, generator expressions, itertools (islice, chain, tee), and yield from for composing stages.

Clarifying questions expected:

Does the output need to be ordered, or can stages run in parallel (which generators alone do not give you)?
Will the data be read once or several times (one-shot iterators vs. re-reading the source)?

Code / implementation expected: Yes — a three- or four-stage generator pipeline over a file-like source, ideally showing that items flow one at a time.

generators
pipeline

Try it yourself

Sketch your approach before you read the solution. That is what the interview tests.

Stuck? Get a hint.

A short nudge toward the approach. It will not give the answer away.

Solution

Target Audience: Developers who know basic generators and want to process big or endless data streams; candidates preparing for "how would you process a huge file?" questions. Difficulty: Medium Python version: Behaviour checked on CPython 3.11.17 and in the page runner (Pyodide 0.25.1, CPython 3.11.3).

How to read this doc: Each idea is explained in plain words first, then tagged with 📌 Interview term: — that is the exact vocabulary an interviewer expects to hear. Every output shown was printed by a real run; the Verified section quotes it word for word.

1. Why This Even Matters — A Story First

Think of a sushi conveyor belt versus a buffet. At a buffet, the kitchen cooks everything first and lays it all out; you need a huge table and lots of food sits around. On a conveyor belt, the chef makes a plate only when there is room on the belt, each plate pass