Aggregation is where you turn many rows into a summary: total revenue per store, average rating per film, number of rides per day. In Spark it's groupBy followed by agg.
The shape is always the same — group by some keys, then compute one or more aggregates:
from pyspark.sql import functions as F
orders.groupBy("store_id").agg(
F.sum("amount").alias("total_revenue"),
F.avg("amount").alias("avg_order"),
F.count("*").alias("num_orders"),
)
A few things worth knowing:
Always alias your aggregates. Without .alias(...), Spark names the column something like sum(amount), which is ugly and hard to reference later. Name them.
count("*") vs count("col"). count("*") counts rows; count("col") counts non-null values in that column. That difference matters when data has nulls — and interviewers love to test it.
Common aggregate functions: sum, avg (mean), min, max, count, countDistinct, first, last, collect_list, collect_set. They all live in pyspark.sql.functions.
Grouping by multiple keys is just more columns: groupBy("store_id", "day"). You get one row per unique combination.
Filtering after aggregation (SQL's HAVING) is just a filter on the aggregated DataFrame:
orders.groupBy("store_id") \
.agg(F.sum("amount").alias("total_revenue")) \
.filter(F.col("total_revenue") > 1000)
That's the whole pattern: group, aggregate, optionally filter. One subtlety to keep in your pocket — when you compare aggregated numbers (averages, ratios), tiny floating-point differences are normal, so Spark-Plug grades those with a small tolerance rather than demanding an exact match.
Practice: daily-ride-count (count per group) and barista-averages (avg with a HAVING-style filter).