The shift manager at The Daily Grind wants a list of the large drinks. You write the filter, it runs without a murmur, and the list comes back with every drink on it. Smalls, mediums, the lot.
orders.filter(F.col("size") == "large")
result = orders
Nothing is broken. This is Spark doing precisely what you asked, and the reason why is the one idea that makes the rest of the DataFrame API click.
Watch it happen
Here are five rows of orders:
| order_id | drink | size |
|---|---|---|
| 1001 | oat latte | large |
| 1002 | drip coffee | small |
| 1003 | caramel macchiato | large |
| 1004 | flat white | medium |
| 1005 | cold brew | large |
filter returns a new DataFrame and leaves orders exactly as it was. The code above threw that new DataFrame away and handed the grader the original: all five rows. Keep hold of it instead:
result = orders.filter(F.col("size") == "large")
| order_id | drink | size |
|---|---|---|
| 1001 | oat latte | large |
| 1003 | caramel macchiato | large |
| 1005 | cold brew | large |
orders still has five rows. It always will. No transformation can change a DataFrame; it can only describe a new one.
A DataFrame is a description, not a box of rows
That is the whole idea. orders doesn't hold five rows in memory; it holds a description of where rows come from ("the orders fixture"). orders.filter(...) holds a slightly longer description ("the orders fixture, then keep the large ones"). Nothing has been filtered. Spark writes down what you asked for and does no work until something demands actual rows. This is lazy evaluation.
So every DataFrame operation is one of two kinds:
- Transformations extend the description and return a new DataFrame:
select,filter,withColumn,join,orderBy,groupBy(...).agg(...). They return instantly because they never touch data. - Actions demand rows:
show(),count(),collect(),first(). The moment you call one, Spark takes the whole description, optimises it, and runs it.
Why bother being lazy? Because seeing the whole description before doing any work lets Spark rearrange it. A filter you wrote last can run first, so every step after it handles fewer rows. You write the logic in the order that reads well; Spark picks the order that runs well.
Feel it
In the practice problem below, press Run before you change anything. The starter code uses a column called paid that was never kept, and the error does not point at the line that uses it. It arrives when the grader calls the action, complaining that paid is missing from the schema, because that is the first moment any of your lines is actually checked. You can even build a DataFrame from a column that doesn't exist and, as long as nothing ever asks it for rows, hear no complaint at all.
Where your code actually runs
In every Spark Plug problem the input DataFrames and a spark session are already in scope. You chain transformations, assign the final DataFrame to result, and the grader calls the action and compares the rows. You never call collect() yourself, and on real data you mostly shouldn't: it pulls every row into Python memory. Prefer show() while exploring; its output appears in the results pane.
Your code runs on Sail, not on Apache Spark. Sail is a JVM-free engine that speaks Spark Connect, the protocol PySpark uses to talk to a server, so what you write here is standard PySpark and the same code runs unchanged on a Spark cluster. Everything in this lesson holds on both. What Sail can't show you is a cluster: it runs as a single process, so when a later lesson explains how Spark spreads work across many machines, that part is explained with diagrams rather than demonstrated, and the lesson will say so.
Now go and fix the colleague's code: The Filter That Did Nothing.