Encrypted file exchange
Manifest-driven encryption, vendor-specific delivery, and inbound decryption pipelines using Python, shell, and AWS Batch.
The system at a glance.
A conceptual view of the flow and its boundaries.
The challenge.
Large vendor files require encryption, credential isolation, lifecycle tracking, and clear outcomes without forcing binary-heavy work into a short-lived request handler.
My contribution.
Worked across Python/Bash file processing, cloud batch execution, vendor integrations, and encryption-related operational monitoring.
Implementation.
- Use queued events and Lambda triggers to submit isolated encryption jobs to AWS Batch.
- Retrieve vendor public keys from Secrets Manager and encrypt file bytes according to S3 manifests.
- Track manifests through pending, processing, processed, and failed states and publish outcome events.
- Support separate inbound decryption pipelines and key-expiration warning checks.
Design decisions.
Batch containers support GPG binaries and file I/O, with more startup and infrastructure overhead than a pure Lambda path.
One encryption job per message simplifies isolation while increasing per-file scheduling overhead.
Validation.
Unit tests and acceptance scenarios cover file processing, job outcomes and lifecycle transitions.
The outcome.
Provided traceable encryption and decryption paths around external file exchange.
Explore the pattern.
The companion repository is an independent educational demonstration built with synthetic examples. It illustrates related engineering patterns and does not contain employer source code or production data. Repositories are currently private; please contact me for access.
Open private GitHub demo