Everything you have written in this track has a twin in SQL, and Spark runs both through the same planner: spark.sql("SELECT …") returns a DataFrame, and the plan it prints is the one you would get from groupBy and agg. Working data engineers arrive from SQL more often than from pandas, and most teams keep both dialects in the same repository, so the last lesson is about reading the other one.
The same question, twice
The Daily Grind's revenue per store, only the stores above £1,000:
orders.groupBy("store").agg(F.sum("amount").alias("revenue")).filter(F.col("revenue") > 1000)
SELECT store, SUM(amount) AS revenue
FROM orders
GROUP BY store
HAVING SUM(amount) > 1000
Read them side by side and the track is in the SQL. filter before a groupBy is WHERE; after it, HAVING. select(...).alias(...) is SELECT … AS. join(on=, how="left") is LEFT JOIN … ON. And the window lessons:
ROW_NUMBER() OVER (PARTITION BY genre ORDER BY plays DESC, title ASC) AS rn
is Window.partitionBy("genre").orderBy(...) with F.row_number().over(w), word for word, in the other order. rowsBetween(-2, 0) is ROWS BETWEEN 2 PRECEDING AND CURRENT ROW, and the default frame is the same RANGE in both, with the same tie-lumping.
Which to write
SQL reads best when the question is one statement: a few joins and an aggregate that an analyst can read top to bottom. The DataFrame API reads best when the logic is built in pieces: a filter decided by a parameter, a list of columns from a loop, a transformation you want to unit-test as a Python function. Neither is faster; the planner sees the same thing. Most real pipelines are SQL for the shape and Python for the glue.
What this site runs
This site runs the DataFrame API only. The sandbox that runs your code does not accept spark.sql, because a SQL string can name files and create functions, and the sandbox's whole job is to make sure your code cannot. Where the SQL twin is shown here, it is to read, not to run. If you want the reps, the DataFrame problems are the reps; the SQL comes free once you can read it.
Self-check
Which of these is the DataFrame twin of HAVING COUNT(*) > 1?
.filter(F.count("*") > 1)before thegroupBy.agg(F.count("*").alias("n")).filter(F.col("n") > 1).where("count > 1")on the input table
The second. HAVING is a filter on an aggregate, so it has to come after the agg, on the aliased column: the One Row Per Group lesson's "keep only some groups".
That is Part 1. Fifteen lessons, forty-four graded problems, every claim measured on the engine you ran them on. What you can now read: a plan, a window, a join that multiplied, a null that vanished, a job's stages, and a SQL query someone else wrote. Every Spark term the track uses is in the glossary, with Spark's own definition beside it. Part 2 is the performance track: concept lessons on why a job is slow and what to do about it, starting with Reading a Plan.