Database operations
PostgreSQL integration and retention jobs, plus warehouse provisioning and operational monitoring.
The system at a glance.
A conceptual view of the flow and its boundaries.
The challenge.
Application data needs a reliable path into relational systems, clear retention behavior, and infrastructure that operators can provision and monitor consistently.
My contribution.
Worked across Python/SQL database integration, scheduled data maintenance, and OpenTofu-based warehouse setup.
Implementation.
- Process queued contact events and upsert contact records and related entities into PostgreSQL.
- Manage database access through Secrets Manager-backed credentials and VPC-connected Lambda functions.
- Implement scheduled retention processing and server-side PostgreSQL-to-S3 export patterns.
- Provision Redshift clusters/workgroups, Glue catalog resources, DNS, and CPU/disk/connection health alarms.
Design decisions.
Connection-per-contact and child-record replacement simplify individual operations but require attention to connection cost and transaction boundaries.
Provisioning analytics infrastructure and maintaining the application data flow are separate operational responsibilities.
Validation.
Mocked unit tests and acceptance scenarios cover integration and retention logic. Database integration tests require a connected environment.
The outcome.
Connected event-driven application data with relational storage, lifecycle jobs, and monitored analytics infrastructure.
Explore the pattern.
The companion repository is an independent educational demonstration built with synthetic examples. It illustrates related engineering patterns and does not contain employer source code or production data. Repositories are currently private; please contact me for access.
Open private GitHub demo