Event-driven orchestration
A stateful orchestration layer connecting object-storage events to ordered Airflow and Databricks processing.
The system at a glance.
A conceptual view of the flow and its boundaries.
The challenge.
File-driven workloads need ordering, duplicate protection, and a clear recovery path when a downstream job fails. A trigger alone does not provide an operationally complete pipeline.
My contribution.
Owned Python orchestration and infrastructure connecting file staging, queued dispatch, workflow execution, and operational state tracking.
Implementation.
- Stage input files before dispatch and record lifecycle transitions in DynamoDB.
- Use SQS FIFO, single-record dispatch, and conditional state updates for ordering and duplicate protection.
- Trigger MWAA DAGs that invoke Databricks jobs and separate successful processing from failed-file recovery.
- Provide CloudWatch operational visibility and SNS failure notifications alongside unit, acceptance, and end-to-end validation.
Design decisions.
Serial execution makes ordering and recovery easier to reason about, at the cost of throughput when arrivals exceed processing capacity.
A long queue visibility window accommodates data jobs but delays retry after genuine execution failures.
Validation.
Unit tests isolate AWS boundaries; acceptance scenarios and end-to-end checks exercise file processing and recovery paths.
The outcome.
Created an auditable processing path with explicit staging, queued, running, success, and failure states.
Explore the pattern.
The companion repository is an independent educational demonstration built with synthetic examples. It illustrates related engineering patterns and does not contain employer source code or production data. Repositories are currently private; please contact me for access.
Open private GitHub demo