Contact history & quality
PySpark normalization, profile enrichment, rule-driven quality checks, and Iceberg-backed contact history.
The system at a glance.
A conceptual view of the flow and its boundaries.
The challenge.
Sent, open, click, bounce, holdout, and selection events arrive in different formats and need a consistent history model with repeatable validation.
My contribution.
Worked across event normalization, Spark SQL transformations, data quality rules, and lakehouse data-product operations.
Implementation.
- Split engagement exports into semantic event datasets and enrich them using profile lookup tables.
- Transform event records into a unified communication-history model and write affected Iceberg partitions.
- Use S3 completion markers to support reruns, with processed-file movement and Iceberg snapshot/orphan maintenance.
- Apply configurable null, blank, duplicate, count-trend, and custom SQL checks through reusable PySpark quality tooling.
Design decisions.
Partition-based processing keeps batch outputs predictable and makes the reprocessing boundary explicit.
Bounded lookback windows make reprocessing manageable but require a separate approach for older late-arriving records.
Validation.
Runtime data-quality rules and reconciliation notebooks check completeness, consistency and expected transformations.
The outcome.
Created a normalized data foundation for communication-history reporting and downstream integration.
Explore the pattern.
The companion repository is an independent educational demonstration built with synthetic examples. It illustrates related engineering patterns and does not contain employer source code or production data. Repositories are currently private; please contact me for access.
Open private GitHub demo