Configure Fivetran Data Pipelines

The skill configures Fivetran data connectors across sources and destinations, either through the Fivetran REST API in Python or the Fivetran Terraform provider for infrastructure-as-code setups. Its core principles: choose incremental vs. full-refresh sync based on source capability and data volume, never expose database credentials directly (SSH tunneling, VPN, IP whitelisting, or Fivetran's own credential storage instead), and set sync frequency to match data-freshness needs without over-syncing or straining the source system.

Use it to stand up a new connector (Postgres, Salesforce, a custom REST API), configure its sync settings, or manage the whole setup as Terraform rather than through Fivetran's UI. A representative Terraform connector resource:

# Create PostgreSQL connector
resource "fivetran_connector" "postgres_prod" {
  group_id = fivetran_group.analytics_group.id
  service  = "postgres"
  
  destination_schema {
    name = "postgres_production"
  }
  
  config = {
    host     = var.postgres_host
    port     = 5432
    database = "production"
    user     = var.postgres_user
    password = var.postgres_password
    
    connection_method = "SSH"
    tunnel_host      = var.bastion_host
    tunnel_port      = 22
    tunnel_user      = var.ssh_user
  }
  
  sync_frequency = 60  # 1 hour
}

It is not a general ETL/transformation skill - schema evolution and column-level control (enabling tables/columns, hashing PII) are covered, but downstream transformation logic in the warehouse is out of scope.

Input is the source/destination pair and its connection details. Output is a working connector configuration - either a Fivetran API payload (service, group_id, per-source config like SSH tunnel host/port/user) or Terraform resources (fivetran_destination, fivetran_connector, fivetran_connector_schema_config) that also control per-column settings such as hashing a PII field like email. Custom REST API sources are configured with cursor-based pagination, a primary key and a replication key per endpoint; SaaS sources like Salesforce get connector-specific tuning such as Bulk API and PK chunking for large tables.

It targets the Fivetran API/Terraform provider directly, with connectors for Postgres, Salesforce, custom REST APIs and others, syncing into destinations like Snowflake and BigQuery. It also configures webhook-based monitoring - a URL notified on sync_start/sync_end/sync_failed/schema_change events - and recommends destination-specific tuning, such as Snowflake's COPY command or BigQuery streaming inserts, plus tiered sync frequencies (hourly for critical data, weekly for archival).

Data engineers managing Fivetran pipelines who want connectors configured securely and reproducibly - via API scripts or Terraform - rather than manual UI setup, with PII hashing, incremental sync tuning and monitoring built in from the start.