Practice / strings dates nulls

The Phantom Watchers

hard

BingeBox, a streaming service, has a data-quality poltergeist. Some rows in the watch logs have a null minutes_watched (the player crashed before reporting) and some have a null device (ancient smart TVs that predate the tracking SDK). The analytics team calls them the phantom watchers — and right now the phantoms are silently vanishing from every report, because comparing anything to null in Spark yields null, and a null filter condition drops the row.

Your job: exorcise the nulls without losing the rows.

The watch_logs DataFrame:

Your task — produce a cleaned engagement report:

  1. minutes_clean: minutes_watched with nulls replaced by 0.0.
  2. device_clean: device with nulls replaced by "unknown".
  3. engagement: derived from minutes_clean
    • "binge" when minutes_clean >= 60
    • "casual" when minutes_clean >= 10 (but under 60)
    • "phantom" otherwise (under 10 — including the resurrected zero-minute rows)
  4. Keep only rows where the engagement is "binge" or "phantom" — the two groups the retention team is studying. A phantom row with a null device must survive all the way to the output.
  5. Return exactly log_id, show, device_clean, engagement.

Assign the DataFrame to result. Row order doesn't matter.

watch_logsinput DataFrame

Schema
columntype
log_idlong
showstring
minutes_watcheddouble
devicestring
Sample rows
log_idshowminutes_watcheddevice
501Crab Lawyer47mobile
502Time Sheriff92tv
503Ghost Auditorsnullnull
504Neon Bakers5mobile
505Crab Lawyer60null
506Time Sheriffnulltablet
507Deep Sea Diners30tv
508Neon Bakers120mobile
509Slow Melt10tv
Expected output shape
log_idshowdevice_cleanengagement· 6 rows
Hint

F.coalesce(F.col("minutes_watched"), F.lit(0)) turns nulls into zeros. Build minutes_clean and device_clean first with withColumn, then derive engagement from minutes_clean using chained F.when(...).when(...).otherwise(...). Filter last — on the cleaned columns, never the raw nullable ones (comparisons against null are null, and a null condition drops the row).

Lesson refresher

This problem builds on Selecting & Filtering (~6 min). Pop it open in a new tab if you want a quick recap.

Loading editor…
Hit Run to execute your code and see the output here.