Joins

Joins combine two DataFrames on a matching key: orders with the customers who placed them, plays with the tracks that were played. If you know SQL joins, PySpark joins are the same ideas with a slightly different spelling.

The basic form:

orders.join(customers, on="customer_id", how="inner")

on is the key (a column name, or a list of names, or a condition), and how is the join type.

The join types, and when to use them:

• inner (default) — only rows that match on both sides. The safe workhorse.
• left (a.k.a. left_outer) — all rows from the left, matched where possible, nulls where not. Use when the left side is your "spine" and you want to keep every row.
• right — the mirror image; less common (usually you just swap the DataFrames and use left).
• full (outer) — everything from both sides, nulls where either doesn't match.
• left_semi — rows from the left that have a match, but no columns from the right. This is "filter left by existence in right."
• left_anti — rows from the left that have no match. This is "find the missing ones" — customers with no orders, users who never logged in. Extremely handy.

Watch out for these:

Duplicate column names. If both sides have a column that isn't the join key, you'll get two columns with the same name and ambiguity errors. Join on a shared key name (on="customer_id") rather than a condition when you can — Spark collapses the key into one column for you.

Fan-out. If the right side has multiple rows per key, an inner/left join multiplies the left rows. Sometimes that's what you want; sometimes it silently inflates your counts. Know your key's cardinality.

Big-small joins. When one side is small, a broadcast hint (F.broadcast(small_df)) tells Spark to ship the small side to every worker and skip an expensive shuffle. Good instinct to build early.

Practice: who-ordered-what (inner join) and ghosted-customers (left anti join).