Build Robust Data Pipelines
A data engineering bundle for building ETL pipelines from OLTP sources into an analytics warehouse with monitoring.
1.0.0Add to Favorites
Why it matters
Automate the extraction, transformation, and loading of data into analytical databases. This bundle empowers data engineers and analysts to build reliable data processing pipelines for real-time analytics and optimized querying.
Outcomes
What it gets done
Orchestrate ETL processes using Airflow DAGs
Implement Change Data Capture for incremental data loading
Optimize database queries and table partitioning
Monitor data quality and pipeline performance
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/vb-data-engineering | bash Overview
Data Engineering
What it does
This bundle builds a full ETL workflow: defining sources, orchestrating extraction and incremental loading with Airflow, capturing ongoing changes, and optimizing the resulting warehouse queries and schemas.
How it connects
Use it for ongoing ETL/ELT pipeline work moving data from OLTP sources into an analytics warehouse on a schedule; skip it for pure local exploration with no pipeline or scheduling needs.
Source README
Who This Bundle Is For
Data engineers and analysts building data processing pipelines.
What's Included
MCP Servers
PostgreSQL - OLTP database. Transactions, data source.
ClickHouse - OLAP database for analytics. Fast aggregations on large datasets.
SQLite - lightweight database for local development and testing.
Airflow - pipeline orchestration. DAGs, scheduling, monitoring.
Skills
Airflow DAG Builder - create DAGs for task orchestration.
Change Data Capture - capture changes from sources.
BigQuery Partitioning - optimize table partitioning.
Agents
Data Engineer - build reliable data pipelines.
Database Optimizer - optimize queries and schemas.
Analytics Reporter - create analytical reports.
How to Use
- Define your data sources
- Create a DAG for ETL processes
- Set up CDC for incremental loading
- Optimize queries with Database Optimizer
Example Prompt
Create an Airflow DAG for an ETL pipeline:
- Source: PostgreSQL (orders, products, users)
- Sink: ClickHouse (data warehouse)
- Schedule: every hour
- Logic: incremental loading by updated_at
- Alerts: Slack on errors
Data Pipeline Architecture
┌────────────┐ ┌────────────┐ ┌────────────┐
│ PostgreSQL │ │ MySQL │ │ API │
│ (OLTP) │ │ (OLTP) │ │ Sources │
└─────┬──────┘ └─────┬──────┘ └─────┬──────┘
│ │ │
└──────────────────┼──────────────────┘
│
┌──────▼──────┐
│ Airflow │
│ (Extract) │
└──────┬──────┘
│
┌──────▼──────┐
│ Transform │
│ (dbt) │
└──────┬──────┘
│
┌──────▼──────┐
│ ClickHouse │
│ (OLAP) │
└──────┬──────┘
│
┌──────▼──────┐
│ Dashboards │
│ (Metabase) │
└─────────────┘
Results
- Reliable data pipelines
- Real-time analytics
- Optimized queries
- Data quality monitoring
Bundle Contents
This bundle includes: 4 MCP servers, 3 skills, 3 agents
MCP server for PostgreSQL database operations with schema inspection and query execution.
chDB is an in-process SQL OLAP engine powered by ClickHouse, with AI-agent skills that teach Claude Code and Cursor to write correct chdb code.
Query, write, and inspect SQLite database schemas with Claude - read-only mode available for untrusted databases.
A Model Context Protocol (MCP) server implementation for Apache Airflow, enabling seamless integration with MCP clients. This project provides a standardized way to interact with Apache Airflow through the Model Context Protocol.
AI skill for building robust Apache Airflow DAGs - TaskFlow API, error handling, data quality checks, and monitoring.
AI skill for Change Data Capture systems - log-based CDC via Debezium/Kafka, schema evolution, and database-specific configuration.
Provides expert recommendations on BigQuery table partitioning strategies, optimization techniques, and performance best practices.
Autonomously designs and implements scalable data pipelines, ETL processes, and data warehouse architectures with optimal performance and reliability.
Autonomously analyzes, optimizes, and tunes SQL/NoSQL databases, caching strategies, and data pipelines for maximum performance.
Autonomously analyzes data files, extracts key metrics, and generates comprehensive reports with insights and recommendations.
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.