BingeBox, a streaming service, has a data-quality poltergeist. Some rows in the watch logs have a null minutes_watched (the player crashed before reporting) and some have a null device (ancient smart TVs that predate the tracking SDK). The analytics team calls them the phantom watchers — and right now the phantoms are silently vanishing from every report, because comparing anything to null in Spark yields null, and a null filter condition drops the row.
Your job: exorcise the nulls without losing the rows.
The watch_logs DataFrame:
Your task — produce a cleaned engagement report:
minutes_clean:minutes_watchedwith nulls replaced by0.0.device_clean:devicewith nulls replaced by"unknown".engagement: derived fromminutes_clean—"binge"whenminutes_clean >= 60"casual"whenminutes_clean >= 10(but under 60)"phantom"otherwise (under 10 — including the resurrected zero-minute rows)
- Keep only rows where the
engagementis"binge"or"phantom"— the two groups the retention team is studying. A phantom row with a null device must survive all the way to the output. - Return exactly
log_id,show,device_clean,engagement.
Assign the DataFrame to result. Row order doesn't matter.