Learn / Glossary

Spark glossary

The lessons use plain words first. These are the words Spark's own documentation uses for the same things, quoted, with the lesson that teaches each one, for the day you meet them in a log, a stack trace or the Spark UI.

Driver program: “The process running the main() function of the application and creating the SparkContext.”; Source: Cluster mode overview Taught in: Who Does What on a Cluster
Cluster manager: “An external service for acquiring resources on the cluster (e.g. standalone manager, YARN, Kubernetes).”; Source: Cluster mode overview Taught in: Who Does What on a Cluster
Worker node: “Any node that can run application code in the cluster.”; Source: Cluster mode overview Taught in: Who Does What on a Cluster
Executor: “A process launched for an application on a worker node, that runs tasks and keeps data in memory or disk storage across them. Each application has its own executors.”; Source: Cluster mode overview Taught in: Who Does What on a Cluster
Task: “A unit of work that will be sent to one executor.” One task processes one data partition.; Source: Cluster mode overview Taught in: Who Does What on a Cluster
Job: “A parallel computation consisting of multiple tasks that gets spawned in response to a Spark action (e.g. save, collect).”; Source: Cluster mode overview Taught in: Who Does What on a Cluster
Stage: “Each job gets divided into smaller sets of tasks called stages that depend on each other.” A job is cut into stages at every shuffle.; Source: Cluster mode overview Taught in: Who Does What on a Cluster
Transformation / action: “Transformations, which create a new dataset from an existing one, and actions, which return a value to the driver program after running a computation.” Transformations are lazy; an action runs the plan.; Source: RDD programming guide Taught in: Nothing Runs Until You Ask
Data partition: The chunk of rows a single task processes: “a single task will operate on a single partition.” The track says data partition to keep it apart from a window partition.; Source: RDD programming guide Taught in: Where Your Code Runs
Shuffle: “Spark’s mechanism for re-distributing data so that it’s grouped differently across partitions. This typically involves copying data across executors and machines, making the shuffle a complex and costly operation.” groupBy, distinct and join cause one; filter, select and withColumn never do.; Source: RDD programming guide Taught in: Where Your Code Runs
Broadcast join: A join where the small side is shipped whole to every executor so no shuffle is needed. Automatic under spark.sql.autoBroadcastJoinThreshold (10 MB by default); F.broadcast(df) asks for it “even if the size of table … is above the configuration”.; Source: Performance tuning Taught in: How a Join Is Planned
Join strategy hint: “When different join strategy hints are specified on both sides of a join, Spark prioritizes the BROADCAST hint over the MERGE hint over the SHUFFLE_HASH hint over the SHUFFLE_REPLICATE_NL hint.” The four names are the four strategies: broadcast hash, sort-merge, shuffle hash, and nested loop.; Source: Performance tuning Taught in: How a Join Is Planned
Skew: One key far more common than the rest, so one data partition holds most of the rows and one task does nearly all the work. Spark’s adaptive execution “dynamically handles skew in sort-merge join by splitting (and replicating if needed) skewed tasks into roughly evenly sized tasks”; a skewed groupBy is yours to salt.; Source: Performance tuning Taught in: When One Key Has Everyone
Salt: A small number added to a hot key, derived from something that varies, so the rows sharing that key spread across several data partitions instead of one. Costs a second pass to combine the salted groups. The track’s term; Spark’s docs describe the problem, not this fix.; Source: Performance tuning Taught in: When One Key Has Everyone
Adaptive query execution: “An optimization technique in Spark SQL that makes use of the runtime statistics to choose the most efficient query execution plan, which is enabled by default since Apache Spark 3.2.0.” It coalesces post-shuffle partitions, turns a sort-merge join into a broadcast join, and splits skewed join tasks.; Source: Performance tuning Taught in: How Many Pieces
Narrow dependency: What coalesce has and repartition does not: “this operation results in a narrow dependency, e.g. if you go from 1000 partitions to 100 partitions, there will not be a shuffle, instead each of the 100 new partitions will claim 10 of the current partitions.”; Source: DataFrame.coalesce Taught in: How Many Pieces
Storage level: How a cached dataset is kept. “The cache() method is a shorthand for using the default storage level, which is StorageLevel.MEMORY_ONLY” for an RDD; PySpark’s DataFrame.cache() “Persists the DataFrame with the default storage level (MEMORY_AND_DISK_DESER)”, so partitions that do not fit in memory go to local disk. unpersist() gives the memory back.; Source: RDD programming guide Taught in: Keep It Around
Unified memory region: “Execution memory refers to that used for computation in shuffles, joins, sorts and aggregations, while storage memory refers to that used for caching … In Spark, execution and storage share a unified region (M).” 60% of the heap after 300 MiB by default (spark.memory.fraction), half of it cache that execution cannot evict (spark.memory.storageFraction).; Source: Tuning guide Taught in: Where the Memory Goes
Spill: Execution memory running out and going to disk, shown per task in the Stages tab as “Shuffle spill (memory)”, “the size of the deserialized form of the shuffled data in memory”, and “Shuffle spill (disk)”, “the size of the serialized form of the data on disk”. Not an error; the same causes as an executor out-of-memory, one step earlier.; Source: Web UI Taught in: Where the Memory Goes
Predicate pushdown: A filter moved into the file scan so chunks it rules out are never read. For Parquet, spark.sql.parquet.filterPushdown “enables Parquet filter push-down optimization when set to true” and is true by default.; Source: Parquet files Taught in: What a File Is For
Partition discovery and pruning: “In a partitioned table, data are usually stored in different directories, with partitioning column values encoded in the path of each partition directory.” Spark discovers the column from the paths, and a filter on it opens only the directories that match. A directory partition, not a data partition and not a window partition.; Source: Parquet files Taught in: What a File Is For
Spark UI: “Apache Spark provides a suite of web user interfaces (UIs) that you can use to monitor the status and resource consumption of your Spark cluster.” Jobs, Stages, Storage, SQL and Executors tabs, served by the driver on port 4040 by default.; Source: Web UI Taught in: Where to Look When It's Slow
Column pruning: The optimiser reading only the columns a query’s result needs, whatever the code selected: TableScan projection=[size] in an optimised plan for a groupBy on size. The track’s term for what explain("extended") shows between the parsed and optimised plans.; Source: DataFrame.explain Taught in: Reading a Plan
Window partition: The group of rows a window function treats together: Window.partitionBy. Not a data partition — the same word for a different thing, which is why the track always says which.; Source: Window API Taught in: Ranking Within Groups
Window frame: The rows a window function can see from the current row. “When ordering is defined, a growing window frame (rangeFrame, unboundedPreceding, currentRow) is used by default.” rowsBetween counts rows; the default RANGE frame reasons by value and treats ties as one step.; Source: Window API Taught in: Running Totals and Neighbours
Competition ranking: What rank() does: tied rows share a rank and the next rank skips the places they took, like a podium. dense_rank() ties without the gap; row_number() never ties.; Source: pyspark.sql.functions.rank Taught in: Ranking Within Groups
ANSI mode: “Since Spark 4.0, spark.sql.ansi.enabled is on by default.” With it on, a cast or parse that cannot succeed is an error, not a null.; Source: SQL migration guide Taught in: A Date Is Not a String
Spark Connect: “A decoupled client-server architecture that allows remote connectivity to Spark clusters using the DataFrame API and unresolved logical plans as the protocol.” The driver lives on the server; your code is the client.; Source: Spark Connect overview Taught in: Who Does What on a Cluster
Sail: The engine under this site: a Spark Connect server, not the JVM, running here in its local mode (one process). “In local mode, Sail runs as a single process.” It also has a cluster mode with a driver and workers, which this site does not use.; Source: Sail documentation Taught in: Nothing Runs Until You Ask