Build Robust Data Pipelines
A bundle for building ETL pipelines with Airflow orchestration, CDC loading, and query optimization.
1.0.0Add to Favorites
Why it matters
Automate the extraction, transformation, and loading of data into analytical databases. This bundle empowers data engineers and analysts to build reliable data processing pipelines for real-time analytics and optimized querying.
Outcomes
What it gets done
Orchestrate ETL processes using Airflow DAGs
Implement Change Data Capture for incremental data loading
Optimize database queries and table partitioning
Monitor data quality and pipeline performance
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/vb-data-engineering | bash Overview
Data Engineering
A bundle for building ETL pipelines: Airflow orchestration, incremental CDC loading, and query optimization into an analytics warehouse. Use it when building or maintaining a scheduled pipeline moving data from transactional sources into an analytics warehouse.
The workflow you build
This bundle assembles a full ETL/ELT pipeline: extract from OLTP sources (like PostgreSQL) with Airflow-orchestrated DAGs, capture incremental changes via CDC, transform and load into a ClickHouse OLAP warehouse for fast aggregations, and optimize the resulting queries and schemas - producing reliable, monitored data pipelines that feed real-time analytics. A typical workflow: define your PostgreSQL/MySQL/API sources, build an hourly incremental-load DAG (e.g. orders/products/users into a ClickHouse warehouse, with Slack alerts on failure), set up change data capture for incremental loading, and hand off to a database optimizer for query and schema tuning.
When to use this bundle - and when NOT to
Use it when you're building or maintaining a data pipeline that needs to move data from transactional sources into an analytics warehouse on a schedule, with orchestration, incremental loading, and query optimization built into the same workflow. It is not meant for one-off ad hoc queries or a single lightweight local database task - SQLite is included specifically for local development and testing, not as the pipeline's production target.
How to get started
Start by defining your data sources, then create an Airflow DAG for the ETL process, set up change data capture for incremental loading, and finish by optimizing queries with the included database-optimization capability. A typical starter prompt: "Create an Airflow DAG for an ETL pipeline: Source: PostgreSQL (orders, products, users), Sink: ClickHouse (data warehouse), Schedule: every hour, Logic: incremental loading by updated_at, Alerts: Slack on errors."
Who it's for
Data engineers and analysts building data processing pipelines who need orchestration, incremental change capture, an analytics-ready warehouse, and query optimization working together rather than assembled from separate ad hoc tools.
Source README
Who This Bundle Is For
For data engineers and analysts building data processing pipelines.
What's Included
MCP Servers
PostgreSQL - OLTP database. Transactions, data source.
ClickHouse - OLAP database for analytics. Fast aggregations on large datasets.
SQLite - lightweight database for local development and testing.
Airflow - pipeline orchestration. DAGs, scheduling, monitoring.
Skills
Airflow DAG Builder - creating DAGs for task orchestration.
Change Data Capture - capturing changes from sources.
BigQuery Partitioning - optimizing table partitioning.
Agents
Data Engineer - building reliable data pipelines.
Database Optimizer - optimizing queries and schemas.
Analytics Reporter - creating analytical reports.
How to Use
- Define your data sources
- Create a DAG for ETL processes
- Set up CDC for incremental loading
- Optimize queries with Database Optimizer
Example Prompt
Create an Airflow DAG for an ETL pipeline:
- Source: PostgreSQL (orders, products, users)
- Sink: ClickHouse (data warehouse)
- Schedule: every hour
- Logic: incremental loading by updated_at
- Alerts: Slack on errors
Data Pipeline Architecture
┌────────────┐ ┌────────────┐ ┌────────────┐
│ PostgreSQL │ │ MySQL │ │ API │
│ (OLTP) │ │ (OLTP) │ │ Sources │
└─────┬──────┘ └─────┬──────┘ └─────┬──────┘
│ │ │
└──────────────────┼──────────────────┘
│
┌──────▼──────┐
│ Airflow │
│ (Extract) │
└──────┬──────┘
│
┌──────▼──────┐
│ Transform │
│ (dbt) │
└──────┬──────┘
│
┌──────▼──────┐
│ ClickHouse │
│ (OLAP) │
└──────┬──────┘
│
┌──────▼──────┐
│ Dashboards │
│ (Metabase) │
└─────────────┘
Results
- Reliable data pipelines
- Real-time analytics
- Optimized queries
- Data quality monitoring
Bundle Contents
This bundle includes: 4 MCP servers, 3 skills, 3 agents
An MCP server giving Claude read-only PostgreSQL access for schema inspection and safe, SELECT-only data analysis.
In-process SQL OLAP engine powered by ClickHouse with a pandas-compatible DataStore API, supporting 60+ file formats including Parquet, CSV, JSON, Arrow, and
Query, write, and inspect SQLite database schemas with Claude - read-only mode available for untrusted databases.
A Model Context Protocol (MCP) server implementation for Apache Airflow, enabling seamless integration with MCP clients. This project provides a standardized way to interact with Apache Airflow through the Model Context Protocol.
AI skill for building robust Apache Airflow DAGs - TaskFlow API, error handling, data quality checks, and monitoring.
AI skill for Change Data Capture systems - log-based CDC via Debezium/Kafka, schema evolution, and database-specific configuration.
A BigQuery partitioning skill covering partition type selection, granularity, clustering strategy, and cost/performance optimization.
Agent for designing and building ETL/ELT data pipelines and data-warehouse solutions with quality and monitoring built in.
An autonomous agent that profiles SQL/NoSQL databases, designs index and caching strategies, and delivers optimization plans with rollback and monitoring.
An autonomous agent that analyzes data files and produces a structured report with metrics, trends, and prioritized recommendations.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.