End-to-end Metadata-Driven Data Engineering framework built on Azure. Features dynamic SQL/REST API ingestion with range pagination, automated schema mapping, and event-driven orchestration. Implements robust CI/CD via GitHub Actions/YAML and automated failure alerting with Logic Apps. Optimized for scalability and DE best practices.

This project implements a comprehensive event-driven data pipeline for e-commerce transactional data processing using Databricks, PySpark, and Delta Lake. The pipeline handles multiple data sources with advanced data engineering patterns including SCD2 (Slowly Changing Dimensions), data validation, enrichment, and automated archiving.

Incremental batch data pipeline using AWS S3 and AWS Glue with date-based partitioning.

E-commerce analytics pipeline: raw → staging → marts, incremental load, soft/hard delete handling, source freshness. 7 models, 16/16 tests.

Snowflake ELT pipeline with incremental load, MERGE logic, and performance optimization

End-to-end data engineering pipeline on Microsoft Fabric: Incremental watermark ingestion implemented from MySQL, PostgreSQL & Google Sheets through Bronze/Silver/Gold medallion architecture to a star schema warehouse, through the Dataflow Gen2.

End-to-end data engineering pipeline using Databricks, Amazon S3, Delta Lake and Unity Catalog, with full and incremental loads.

Production-grade API ingestion pipeline: rate-limited client, watermark-based incremental extraction, staging + MERGE pattern for BigQuery. Airflow orchestration with task groups.

Incremental ETL pipeline (MySQL → StarRocks) with checkpointing, batching, and retry logic to ensure data consistency and fault tolerance.

End-to-end Azure ELT pipeline — ADF dynamic ForEach + Databricks Delta Lake + Snowflake CDC + Incremental watermark load

Metadata-driven incremental ingestion framework using Azure Data Factory, SQL Server, ADLS Gen2, control tables, watermarks, and operational validation evidence.

Production-style data engineering portfolio project for a Legal CRM: API ingestion, Airflow orchestration, Polars transformations, SQL Server medallion architecture, incremental loads, JSON logs and automated tests.

PySpark-based Incremental Load Pipeline that processes only new and changed customer records, reducing ETL execution time and improving data processing efficiency.

Production-grade SQL Server → Snowflake migration pipeline for a US multi-specialty healthcare clinic. Python · pyarrow · COPY INTO · Incremental loads · HIPAA-compliant DDM · 133 tests.