Data Engineer · GCP & AWS
I build petabyte-scale data platforms across AWS and GCP, with streaming, CDC and metadata-driven pipelines that turn raw data into clean, trusted datasets. That makes data ready for AI engineers and data scientists to train and run models on, as well as for the analysts and dashboards that run the business.
fct_architectures
Seven production pipelines across GCP and AWS. Each shows the problem, how data flows end to end, and what changed afterwards.
Files landing in Cloud Storage had to wait for scheduled batch loads or manual triggers before anyone could query them in BigQuery.
Each file is processed and queryable in BigQuery as soon as it lands, with no polling, schedules or manual steps in between.
More than 30 source systems each needed their own ingestion pipeline, which made every new source slow to add and hard to keep consistent.
One framework ingests every source at petabyte scale. Adding a source means adding its metadata, and Airflow runs all loads with the same scheduling, retries and alerting. The same petabyte-scale design also runs on AWS.
More than 500 MySQL tables had to reach Redshift every day for reporting. Hand-writing a pipeline per table would not scale.
Batch jobs ran on fixed schedules whether or not new data had arrived, and failed runs needed someone to step in and restart them.
Processing starts when data arrives. Step Functions handles retries, alerting and failure recovery between layers, which cut manual intervention.
Analytics needed every insert, update and delete from 18+ MSSQL and MySQL databases in the lake, with full history and PII handled under GDPR.
Event data lived in Kafka topics that analysts couldn't query, and it needed both its full history and the ongoing stream in the lake.
Every topic lands as its own queryable Delta table. A bulk load brought in the history and incremental loads keep each table current, across terabytes of event data.
Document data in MongoDB sat outside the lake, so it couldn't be analysed alongside the relational and event data.
Terabytes of MongoDB data now sit in Delta tables on S3, kept current incrementally and queryable next to every other source in the lake.
dim_skills
Everything I use to build, run and govern data platforms.
dim_education
A grounding in data analytics and information technology.
Queen Mary University of London
London, UK
Delhi Technological University
New Delhi, India
mart_contact
Open to Data Engineer roles on AWS or GCP: large-scale ingestion, streaming, CDC, lakehouses and data platforms for AI.