Why Spark, and When Not To

Already write Spark at work? Skip to lesson 2. Nothing here will surprise you.

The Daily Grind began as one café whose till exported a CSV. Someone opened it in pandas, charted the busiest hour, and everyone was delighted. Then the chain grew to four hundred shops, the export became a year of every receipt from every till, and the same read_csv sat for a minute before the kernel died with its memory exhausted.

The question hadn't changed. It was still "when are we busiest?". The data had simply stopped fitting on one machine.

That is the whole reason Spark exists

pandas holds your data in one machine's memory and works through it there. Fast and pleasant until it doesn't fit, and then there is no next step: a bigger laptop is not a plan.

Spark's answer is to describe the work once, in code that looks a lot like pandas, and run it across many machines that each hold a slice of the data. Whether orders is ten rows or ten billion, orders.filter(...) is the same line. Which machine does what is Spark's problem, not yours.

Two panels. In the first, a small orders table fits inside a solid panel labelled one machine's memory, and is labelled fits, pandas. In the second, the same table is taller than the panel: four rows are inside it, and three more rows continue below the panel's bottom edge inside a dashed outline; it is labelled doesn't fit, Spark. Spark is for the second case.

fits · pandasone machine's memory

order_id	drink	size
1001	oat latte	large
1002	drip coffee	small
1003	caramel macchiato	large
1004	flat white	medium

doesn't fit · Sparkone machine's memory

order_id	drink	size
1001	oat latte	large
1002	drip coffee	small
1003	caramel macchiato	large
1004	flat white	medium

1005	cold brew	large
1006	espresso	small
1007	mocha	large

3 rows with nowhere to go

When pandas is the right answer

Most data is small. If yours fits comfortably in your laptop's memory, pandas or DuckDB will be quicker to write and quicker to run. Spark pays a start-up cost on every job and shares out work a single core would have finished first. Reaching for Spark on a 200 MB CSV isn't thorough, it's slow.

Reach for Spark when the data doesn't fit, when a pipeline runs nightly on data that keeps growing, or when your team already runs a Spark platform and the job is written for it. That last one is why most people learn it, and it's a perfectly good reason.

What you'll write here

The DataFrame API: standard PySpark, the same code that runs on a cluster, run here by an engine called Sail that lesson 3 introduces properly.

Now run something real. Long Visits is one filter and one select; its hint shows you the shape. Run, look at the rows, Submit, then come back for lesson 2.