Where the Memory Goes

java.lang.OutOfMemoryError is the error every Spark user meets, and the first question is always whose memory: the driver's, or an executor's. They are different processes with different jobs (Who Does What on a Cluster), and the fix is different for each.

Two kinds of memory in an executor

Spark's tuning guide: "Execution memory refers to that used for computation in shuffles, joins, sorts and aggregations, while storage memory refers to that used for caching and propagating internal data across the cluster. In Spark, execution and storage share a unified region (M). When no execution memory is used, storage can acquire all the available memory and vice versa."

Two settings shape it. spark.memory.fraction is the size of that unified region, default 0.6 of the heap after 300 MiB is set aside; the docs say "the rest of the space (40%) is reserved for user data structures, internal metadata in Spark, and safeguarding against OOM errors." spark.memory.storageFraction, default 0.5, is the part of the region where "cached blocks [are] immune to being evicted by execution." So a shuffle can take memory from the cache, down to that line, and a cache can take memory from shuffles when they are idle.

A stack of bars, each proportional to its share of an executor's heap. Reserved: 300 megabytes, a short bar. User data and Spark internals: 40 percent of the rest. Unified region M, highlighted: 60 percent of the rest, set by spark.memory.fraction. Inside M, two bars: R, storage immune to eviction, 50 percent of M, set by spark.memory.storageFraction, drawn solid; and execution or storage, movable, the rest of M, drawn dashed.

one executor's heap, at Spark's defaults

reserved300 MiB

user data and Spark internals40% of the rest

unified region M60% of the rest · spark.memory.fraction

inside M

R · storage, immune to eviction50% of M · spark.memory.storageFraction

execution ↔ storage, movablethe rest of M, whichever needs it

The driver

The driver holds the plan, the task bookkeeping, and anything you collect. It does not hold the data unless you make it. A driver out-of-memory is almost always a collect(), a toPandas(), or a broadcast of a table that was not small (How a Join Is Planned).

Reading the error

• Driver OOM after collect() or toPandas(): the result does not fit in one process. Write it out, or aggregate first.
• Executor OOM in a shuffle or join stage: one task's partition is too big, which is skew (When One Key Has Everyone), too few partitions (How Many Pieces), or a broadcast that was not small.
• Spill, in the Stages tab's "Shuffle spill (memory)" and "(disk)" columns: not an error, but execution ran out of its share and went to disk. The same causes, one step earlier.
• GC time high on the Executors tab: many small objects churning. Wider partitions, or more memory.

More memory per executor is the last fix, not the first. The three above are shape problems, and a bigger box holds the same badly shaped load a little longer.

What this site shows

None of this is visible here. Sail is not the JVM, and this site runs one process with its own limits. The numbers above are Spark's documented defaults; the picture is the model you need the day the error names a process.

The symptom, and the fix

A job dies with an out-of-memory error. Read which process: a driver error after an action that brings rows back means stop bringing them back; an executor error in a shuffle stage means fix the shape of the work, with the lesson the shape points to. Then, and only then, more memory.

Self-check

toPandas() on a 50 GB DataFrame fails. Which process ran out of memory, and what would fix it?

The driver: toPandas collects every row to it. Aggregate or sample first, or write the result out and read a slice.

Nothing to grade here. Next, What a File Is For: the decision made before your code runs, in how the data was written.