Professional retrospective · Insi / Fiserv
Modernising a hybrid financial-data platform
Batch and streaming engineering, Terraform-managed infrastructure, and a controlled migration from on-premises systems to AWS in a hybrid financial-data estate.
Context and ownership
The platform and my responsibility.
Context
Working through Insi, formerly Meta, I owned data-platform delivery on a Fiserv engagement spanning Snowflake, AWS, GCP and on-premises systems. The work included separate migration and processing-performance initiatives.
Mandate
Modernise data delivery while preserving validation, reconciliation and continuity across cloud and retained on-premises workloads.
What I owned
- Batch and streaming pipelines across Snowflake, AWS and a hybrid estate.
- Ingestion and processing optimisation using PySpark, Kafka and dbt.
- Implementation and operation of AWS DMS for part of Fiserv's card transaction data migration.
- Infrastructure implementation and management through Terraform.
- Orchestration, data quality, observability and production recovery.
Decisions and trade-offs
Why I chose this approach.
A phased migration with AWS DMS and parallel validation
I favoured an initial load followed by change replication with AWS DMS, keeping the old and new processing paths available while we validated each migration wave.
- Alternative considered
- A larger cutover based on periodic extracts.
- Why it fit
- The data included card transactions. Phased delivery let us reconcile results, check downstream behaviour and apply acceptance criteria before moving a workload.
- Trade-off
- Parallel operation meant maintaining two paths during validation. It spread the reconciliation and recovery work across controlled waves rather than concentrating it at one cutover.
Engineering focus
How the work connected architecture and operation.
Validate the migration path
Implemented and operated AWS DMS for part of Fiserv's card transaction data migration from on-premises infrastructure to AWS. Initial loading, ongoing change replication, reconciliation and dual runs supported the phased migration.
Improve cloud and retained workloads
Delivered a controlled migration for 40% of high-volume financial-operations data. Separately optimised workloads retained on premises under PCI DSS constraints.
Manage the platform as repeatable infrastructure
Implemented and managed infrastructure as code with Terraform, supporting repeatable environments, reviewed changes and ongoing platform operations.
Remove processing bottlenecks
Re-engineered real-time and batch workloads, addressing ingestion and processing bottlenecks across the hybrid estate.
Production outcomes
Results with a defined scope.
Real-time processing speed
Improvement from removing ingestion and processing bottlenecks in real-time workloads.
Batch processing speed
Improvement in batch processing workloads.
Financial-operations data migrated
Share of high-volume financial-operations data migrated. The AWS DMS example concerns part of the card transaction data.
Retained on-premises processing speed
Separate optimisation of workloads that remained on premises under PCI DSS constraints.
Measurement context and publication scope
These are professional outcomes described in my career record. Internal data volumes, benchmark windows, cutover records and customer topology are not reproduced. The 40% migration figure and the 17% retained-estate improvement describe separate initiatives.