- Knowledge sharing - Declarative Data Pipelines
Created by gh-md-toc
This project aims at collecting and sharing reference material about declarative data pipelines, that is, what they are, how to design and to build them.
That goes without saying that AI agents/assistants will be used wherever relevant. For instance, to begin with, a starting point/cornerstone is the Databricks AI Dev Kit and related material:
- Spark Declarative Pipelines (SDP) and Databricks Lakeflow Declarative Pipelines (LDP)
- Databricks Asset Bundles (DAB)
Even though the members of the GitHub organization may be employed by some companies, they speak on their personal behalf and do not represent these companies.
- Data Engineering Helpers - Knowledge Sharing - Cheat sheet - Databricks AI Dev Kit
- Data Engineering Helpers - Knowledge Sharing - AI Skills
- Data Engineering Helpers - Knowledge Sharing - JavaScript (JS) world
- Data Engineering Helpers - Knowledge Sharing - Java
- Data Engineering Helpers - Knowledge Sharing - Python
- Data Engineering Helpers - Knowledge Sharing - Spark
- Data Engineering Helpers - Knowledge Sharing - PostgreSQL
- Data Engineering Helpers - Knowledge Sharing - Modern Data Stack (MDS) in a box
- AI Helpers - Knowledge Sharing - AI skills curated
- AI Helpers - Knowledge Sharing - AI skills curated - Overview notebook
- GitHub - Spec-kit
- Martin Fowler's blog Understanding Spec-Driven-Development: Kiro, spec-kit, and Tessl, by Birgitta Böckeler, Oct. 2025
- GitHub - Databricks Solutions - AI Dev Kit
- Data Engineering Helpers - Knowledge Sharing - Cheat sheet - Databricks AI Dev Kit
- Spark Declarative Pipelines (SDP)
- Databricks Asset Bundles (DAB)
- Title: Deploy streaming incremental and batch processing ETLs using Lakeflow Declarative Pipelines in Databricks Lakehouse Medallion Architecture with agentic AI support
- Date: Apr. 2026
- Author: Mikołaj Sędek (Mikołaj Sędek on LinkedIn, Mikołaj Sędek on Medium)
- LinkedIn post
- GitHub - Lakeflow Declarative Pipelines in Databrick Free Edition
- Title: Spec-Driven Development: A Practical Guide for Software and Data Engineering Teams
- Date: May 2026
- Author: Sanjeeb Panda (Sanjeeb Panda on LinkedIn, Sanjeeb Panda on Medium)
- Medium - Spec-Driven Development: A Practical Guide for Software & Data Engineering Teams
- Overview:
The Metadata Driven framework for Databricks Lakeflow Declarative Pipelines (LDP) (formerly Delta Live Tables (DLT)). Metadata framework that generates production ready Pyspark code for Lakeflow Declarative Pipelines
- Authors/main contributors:
- Mehdi Modarressi (Mehdi Modarressi on LinkedIn, Mehdi Modarressi on GitHub)
- David O'Keefe (David O'Keefe on LinkedIn, David O'Keefe on GitHub, David O'Keefe on Medium)
- GitHub - Lakehouse Plumber
- Author/main contributor: Yasar Kocyigit
- LinkedIn posts:
- GitHub - Metadata-Driven Lakehouse Ingestion
- Overview:
In my initial experiments to democratize onboarding in metadata-driven development (MDD) architectures, the process has been much easier with Databricks AI Dev Kit.
I first built the infra foundation with DAB + MDD, then integrated AI Dev Kit and tested it at Databricks app level.
Technically, what we added:
- Chat-based onboarding entrypoint
- AI Agent runtime for intent-driven orchestration
- MCP layer for Databricks tool execution
- Lakebase for conversation and project-state persistence
- Git-based metadata flow (YAML + DAB)
- CI/CD gates for validation and controlled deployment
Impact so far:
- Faster source/table onboarding
- More consistent metadata-driven pipelines
- Clear path from request -> config -> validation -> deployment
Also, development can be managed via CLI + Claude I tested Codex and Gemini Pro 3.1 as well; for this workflow Claude 4.6 performed best in my experience.