Real data is messy. It has missing values, dates stored as strings, and text that needs cleaning. This lesson covers the everyday functions that handle that mess — the ones that show up in almost every job and interview.
Nulls. Missing data is a null, and it behaves specially: any comparison with null is null (not true, not false), so you test with dedicated functions:
from pyspark.sql import functions as F
df.filter(F.col("email").isNull())
df.filter(F.col("email").isNotNull())
To replace nulls, coalesce returns the first non-null of its arguments, and fillna sets defaults:
df.withColumn("region", F.coalesce(F.col("region"), F.lit("unknown")))
df.na.fill({"score": 0})
F.lit(...) wraps a literal constant so it can sit in a column expression — you'll need it constantly.
Conditional logic with when/otherwise is Spark's if/else:
df.withColumn("tier", F.when(F.col("spend") > 100, "gold")
.when(F.col("spend") > 50, "silver")
.otherwise("bronze"))
This pairs beautifully with aggregation — F.sum(F.when(cond, 1).otherwise(0)) counts rows matching a condition.
Strings. The common ones: F.upper, F.lower, F.trim, F.length, F.concat_ws(sep, a, b), F.substring, F.split, and F.regexp_replace. Filtering by text pattern: F.col("name").like("%coffee%") or .rlike(regex).
Dates. If a date is stored as a string, parse it with F.to_date(col, "yyyy-MM-dd") or F.to_timestamp. Then extract parts (F.year, F.month, F.dayofweek), truncate (F.date_trunc("month", col)), or do arithmetic (F.datediff(end, start), F.date_add). Grouping "per month" almost always means to_date → date_trunc or year/month → groupBy.
None of these are hard individually — the skill is reaching for the right one without thinking. That comes from reps.
Practice: plays-per-month (date handling) and the-phantom-watchers (nulls + conditional counting).