Customer data pipelines
Scala and Spark SQL pipelines joining customer profiles, maintaining change history, and publishing partitioned data products.
The system at a glance.
A conceptual view of the flow and its boundaries.
The challenge.
Campaign systems consume data from multiple profile domains. Those records need consistent joins, lifecycle handling, and predictable output layouts for downstream consumers.
My contribution.
Worked across Scala/Spark transformation logic, SQL-based profile enrichment, CDC processing, runtime configuration, and Databricks delivery.
Implementation.
- Join account, location, household, contact, and preference domains into campaign-ready datasets.
- Apply CDC in the campaign-base pipeline to separate active and expired records using date-partitioned Parquet.
- Use restart flags and external configuration to support staged execution and environment-specific paths.
- Package Scala applications as shaded JARs and configure Databricks job clusters, schedules, concurrency, and retry behavior.
Design decisions.
CDC targets changed records, while separate account/household extraction jobs use full overwrite; these are distinct processing strategies.
Provided-scope Spark dependencies reduce package size but couple applications to the Databricks runtime and its JVM library versions.
Validation.
Build validation and source-to-target reconciliation support scheduled batch delivery. Runtime and library compatibility are part of release review.
The outcome.
Produced structured customer data products with explicit lifecycle partitions and reusable configuration-driven execution.
Explore the pattern.
The companion repository is an independent educational demonstration built with synthetic examples. It illustrates related engineering patterns and does not contain employer source code or production data. Repositories are currently private; please contact me for access.
Open private GitHub demo