You don't need to be a performance wizard to write good Spark, but a little intuition saves you from the traps that make jobs crawl. And — because we're honest here — part of that intuition is knowing when not to use Spark at all.
The shuffle is the thing to respect. Most Spark cost comes from shuffles: moving data across the network so that related rows end up together. groupBy, join, distinct, and orderBy all shuffle. Filtering and select don't. You can't avoid shuffles entirely, but you can avoid needless ones: filter early to shrink data before a join, and don't sort unless you actually need sorted output.
explain() is your friend. Call df.explain() and Spark prints its physical plan. You don't need to read every line — just learn to spot Exchange (a shuffle) and BroadcastHashJoin (the cheap join) versus SortMergeJoin (the shuffly one). Seeing the plan demystifies why a job is slow.
Broadcast the small side. When you join a big table to a small one (a lookup table, a dimension), wrap the small one in F.broadcast(...). Spark ships it to every worker and skips the shuffle entirely. This single habit fixes a huge share of slow joins.
Partitioning. Data is split into partitions that Spark processes in parallel. Too few and you underuse your cluster; too many tiny ones and coordination overhead dominates. You'll rarely tune this early, but know the word — when someone says "it's skewed," they mean one partition got most of the data and one worker is doing all the work.
Now the honest part: when Spark is the wrong tool. In 2026, if your data fits comfortably on one machine — say under ~50–100 GB — a single-node engine like DuckDB or Polars will almost always be faster and simpler than Spark, because you skip all the distributed machinery. Spark earns its keep when data is genuinely large, genuinely distributed, or already living in a Spark-based platform. Reaching for a cluster to crunch a 200 MB CSV is a classic over-engineering move. Knowing this makes you more credible with Spark, not less — you use it where it wins.
Practice: revisit any join problem and call .explain() in the editor to see the plan for yourself.