What a File Is For

A year of Daily Grind tills is twelve billion rows. The busiest-hour report needs two columns and one month. Whether Spark reads two columns of one month or every byte of the year is decided before the first line of your code runs, by how the files were written.

Rows or columns

CSV and JSON store rows: every record, every field, in order, and a reader that wants one column reads them all. Parquet stores columns. Spark's docs: "Parquet is a columnar format that is supported by many other data processing systems," and reading and writing it "automatically preserves the schema of the original data." Each column sits together with its type, so a query that needs size and price reads size and price. The column pruning in Reading a Plan only saves anything when the file can skip a column.

Two panels. Row by row, as CSV or JSON: a table of five orders with order_id, drink, size and price, the size and price columns highlighted inside every row. Column by column, as Parquet: four strips, one per column, each holding that column's five values together; the size and price strips are highlighted. A query that needs size and price reads two strips of the columnar file and every row of the row file.

row by row · CSV, JSON · a query for size and price reads everything

order_id	drink	size	price
1001	oat latte	large	6.5
1002	drip coffee	small	3
1003	caramel macchiato	large	7.25
1004	flat white	medium	5
1005	cold brew	large	5.5

column by column · Parquet · the same query reads two strips

order_id10011002100310041005

drinkoat lattedrip coffeecaramel macchiatoflat whitecold brew

sizelargesmalllargemediumlarge

price6.537.2555.5

Pushing the filter down

A Parquet file also keeps a minimum and maximum for each column in each chunk, so a filter on price > 100 can skip a whole chunk whose maximum is 40 without reading it. Spark does this by default: spark.sql.parquet.filterPushdown "enables Parquet filter push-down optimization when set to true", and it is true. The filter you wrote moves into the scan, the optimiser's rewrite from Reading a Plan with a file that can honour it.

Partitioned directories

"In a partitioned table, data are usually stored in different directories, with partitioning column values encoded in the path of each partition directory": orders/day=2026-03-01/…, orders/day=2026-03-02/…. Spark discovers the column from the paths, and a filter on day then opens only the directories that match: partition pruning. Written by day, the busiest-hour report for March reads thirty-one directories out of three hundred and sixty-five and never opens the rest. Written as one file, it reads the year.

What this site shows

Nothing, on purpose. Your inputs here are already DataFrames and the sandbox does not read files. The three ideas, columns, pushdown and pruning, are the ones to carry to the first job that does.

The symptom, and the fix

A report over one month scans a year. Check the format: CSV? Convert to Parquet. Check the layout: one directory? Write it partitioned by the column you always filter on, usually a date. And check the filter is on the partition column as written, not on an expression of it: day = '2026-03-01' prunes; substr(day, 1, 7) = '2026-03' may not.

Self-check

Which of these lets Spark skip data it does not need: (a) a CSV with a header, (b) a Parquet file, (c) Parquet files in day=… directories?

(b) skips columns and chunks; (c) skips whole directories as well; (a) skips nothing.

Nothing to grade here. Last, Where to Look When It's Slow: the screen every symptom in Part 2 was found on.