Sometimes the data's all there — it's just in the wrong shape. Reshaping is how you turn long into wide, wide into long, and many rows into deduplicated ones.
Distinct / dedupe. df.distinct() drops exact duplicate rows. To dedupe on specific columns, df.dropDuplicates(["user_id"]) keeps one row per key (though which one is arbitrary unless you order first — a common gotcha).
Union. Stack two DataFrames with the same schema:
jan.union(feb) # by position — columns must line up
jan.unionByName(feb) # by column name — safer
Prefer unionByName — it matches columns by name so a reordered schema doesn't silently misalign your data.
Explode. When a column holds an array and you want one row per element, explode unpacks it:
from pyspark.sql import functions as F
# tags = ["spark", "python"] -> two rows
df.withColumn("tag", F.explode("tags"))
This is how you flatten nested data — turning a row with a list into several rows, each with one value.
Pivot. Turn distinct row values into columns — a cross-tab. Sales per store per month, with months as columns:
orders.groupBy("store_id") \
.pivot("month") \
.agg(F.sum("amount"))
Pivot is powerful for reports but can explode your column count if the pivoted field has many distinct values, so pivot on low-cardinality fields (months, categories, statuses), not high-cardinality ones (user IDs).
The reverse of pivot — wide back to long — is stack or unpivot, useful when someone hands you a spreadsheet-shaped table and you need it tidy.
Reshaping is less about clever logic and more about knowing these few tools exist so you don't hand-roll them. When data's in the wrong shape, one of distinct, union, explode, or pivot is usually the answer.
Practice: unique-listeners (distinct) and store-month-grid (pivot).