Selecting & Filtering

The shift manager at The Daily Grind wants to know which large drinks earned a decent tip, so she can find out who's making them. The order log has every drink, every size and every tip. Her question is two moves: keep the rows she cares about, then keep the columns she wants to read. Almost everything you will ever do to a DataFrame is some mix of those two.

Watch it happen

Five rows of orders:

Large drinks tipped at least £1:

from pyspark.sql import functions as F

result = (
    orders
    .filter((F.col("size") == "large") & (F.col("tip") >= 1.0))
    .select("order_id", "drink", "tip")
)

Three rows went: two weren't large, and the cold brew was large but tipped 50p. Two columns went: size and price did their work inside the filter and weren't asked for. And the oat latte survived on a tip of exactly £1.00, because "at least" is >=.

filter chooses rows

large = orders.filter(F.col("size") == "large")

F.col("size") == "large" isn't a value, it's a column expression: a description of a test to run against every row. where is the same method under another name; use whichever reads better.

Combine tests with & (and), | (or) and ~ (not), and wrap each test in parentheses. Python binds & more tightly than ==, so without the brackets the pieces are glued together in the wrong order and the run fails before it starts, with CANNOT_CONVERT_COLUMN_INTO_BOOL.

tipped_large = orders.filter((F.col("size") == "large") & (F.col("tip") >= 1.0))

Three more row-pickers you'll reach for constantly:

orders.filter(F.col("drink").isin("oat latte", "flat white"))
orders.filter(F.col("tip").isNotNull())
orders.filter(F.col("drink").startswith("caramel"))

select chooses columns

Strings work for a plain pick. For anything more, pass expressions, and rename with .alias:

result = orders.select(
    "order_id",
    F.col("drink").alias("item"),
    (F.col("price") + F.col("tip")).alias("paid"),
)

The grader matches columns by name, so paid is not the same answer as price + tip left unnamed.

withColumn adds one

When the ask is "keep everything and add a column", that's withColumn, not filter:

flagged = orders.withColumn("is_pricey", F.col("price") > 6)

A comparison is already a boolean column, so no if is needed. Give an existing name and the column is replaced rather than duplicated.

orderBy sorts rows

orders.orderBy("price")                                  # cheapest first
orders.orderBy(F.col("price").desc())                    # priciest first
orders.orderBy(F.col("size"), F.col("tip").desc())       # by size, then biggest tip within it

orderBy returns the same rows in a chosen order; sort is the same method under another name. .desc() on a column expression reverses it, and several columns sort by the first, then by the next within ties. Two things worth knowing early: the grader does not care about row order unless a problem says so, and a sort is one of the most expensive things you can ask a cluster for, so sort for a person to read, not for the engine's benefit.

Chain them

Real answers are a short pipeline, read top to bottom:

result = (
    orders
    .withColumn("is_pricey", F.col("price") > 6)
    .filter(F.col("size") == "large")
    .select("order_id", "drink", "is_pricey")
)

Add a column, keep the large ones, pick three columns. Every step returns a new DataFrame and leaves the one before it alone, which is why a chain like this is safe to read in order. The next lesson explains why that is, and why none of it runs until the grader asks.

Five problems, in the order they build:

1. Long Visits — one filter, one select, one boundary.
2. Large and In Charge — two tests joined with &; the parentheses matter.
3. Rename the Hits — select with .alias and arithmetic.
4. Flag the Long Rides — add a boolean column, keep every row.
5. Tip-Top Baristas — build a column, then filter on it.