Configure and Deploy Airbyte Data Pipelines

The Airbyte Connection Setup Expert Agent covers setting up, configuring, and operating connections in Airbyte end to end. On architecture it explains source-destination pairs as data flows with transformation capabilities, prefers incremental synchronization over full refreshes to minimize resource usage, and calls out schema evolution and failure recovery as design concerns. On data consistency it distinguishes sync modes - Full Refresh, Incremental Append, and Incremental Deduped - and covers configuring cursor fields for incremental syncs and primary keys for deduplication. For deployment it provides a full Docker Compose stack (init, db, webapp, server services with their environment variables and networking) and a Kubernetes deployment (ConfigMap plus a Deployment spec with memory/CPU requests and limits) for the Airbyte server. For connection configuration it gives example JSON/Python payloads: a PostgreSQL source config using CDC replication with the pgoutput plugin, a Snowflake destination config using internal staging and a raw-data schema, and a sync-catalog JSON block wiring a stream to incremental sync mode with a cursorField and append_dedup destination sync mode. For operations it covers a health-check pattern hitting the Airbyte API and checking database connectivity and workspace volumes, a log4j2 logging configuration for console and file appenders, a troubleshooting checklist (network connectivity, authentication, resource limits, schema changes), and performance practices like batch sizing, connection pooling, cursor-field indexing, and adjusting sync schedules. For security it covers secret management, SSL/TLS, network segmentation, patching, and access-log auditing. Finally it shows creating sources and connections programmatically via the Airbyte REST API, including error handling with requests.exceptions.

Use it when standing up a new Airbyte deployment (Docker Compose or Kubernetes), configuring a specific source or destination connector, choosing sync modes and cursor/primary-key settings for a connection, automating connection creation through the Airbyte API, or troubleshooting sync failures, schema-change breakage, or performance issues. It is not for building a custom Airbyte connector from scratch, and it is not a substitute for reviewing the specific connector's own documentation for source- or destination-specific configuration fields - the examples given (PostgreSQL, Snowflake) are illustrative of the pattern, not exhaustive of every connector's schema.

Inputs are source and destination connection parameters (host, credentials, replication method, warehouse/schema for Snowflake-style destinations) and deployment environment variables for Docker Compose or Kubernetes. Outputs are a running Airbyte deployment, configured source/destination/connection objects (via UI or API), a sync catalog defining stream-level sync mode, cursor field, and primary key, and operational artifacts like health-check scripts and logging configuration.

Docker Compose and Kubernetes for deployment, PostgreSQL as an example CDC source using the pgoutput replication plugin, Snowflake as an example destination using internal staging, and the Airbyte REST API (sources/create, connections/create) for programmatic setup.

Data engineers deploying or operating Airbyte who need concrete Docker Compose/Kubernetes manifests, connector configuration examples, and API automation code rather than a conceptual overview of ELT pipelines.