Configure Fivetran Data Pipelines

This skill demonstrates how to configure Fivetran data pipeline connectors using API-based configuration and Terraform infrastructure-as-code approaches. It shows examples of setting up PostgreSQL connectors with SSH tunneling, Salesforce connectors with Bulk API optimization, Snowflake destinations, custom REST API connectors with OAuth2, sync frequency configuration, schema selection, column hashing for PII, and webhook monitoring.

Use this skill when you need to automate Fivetran connector provisioning via API or Terraform, configure incremental sync strategies for large datasets, implement secure database connections through SSH tunnels or bastions, set up custom REST API connectors with OAuth2 and cursor pagination, or optimize sync frequencies and costs across multiple data sources. Use it when implementing column-level hashing for compliance or configuring webhook monitoring for sync events.

Do NOT use this for writing SQL transformations inside your data warehouse (Fivetran dbt transformations are a separate concern), or for designing the data models themselves - this focuses purely on the extraction and loading configuration layer.

You provide your source system details (host, credentials, database names), destination warehouse configuration, security requirements (SSH keys, IP whitelists), sync frequency needs, and schema selection preferences. The skill demonstrates Python API calls, Terraform HCL configurations, JSON payloads for custom connectors, and configuration patterns for sync modes, authentication methods, and performance settings.

The source material includes configuration examples for PostgreSQL databases with SSH tunnel connections, Salesforce connectors with Bulk API and PK chunking settings, Snowflake destinations, BigQuery references, custom REST APIs with OAuth2 authentication and cursor-based pagination, and webhook configurations for monitoring sync events. Examples use Fivetran's API v1 and Terraform provider version 1.0+.

Data engineers building automated ELT pipelines who need to provision connectors programmatically rather than through the UI. DevOps teams managing data infrastructure as code with Terraform. Analytics engineers responsible for optimizing sync costs and frequencies across dozens of sources. Security-focused teams implementing compliant data ingestion with SSH tunneling, credential vaulting, and PII hashing at the connector level.

Here's a concrete example of configuring sync settings via the Fivetran API:

# Update connector sync settings
sync_config = {
    "sync_frequency": 360,  # 6 hours in minutes
    "schedule_type": "auto",
    "daily_sync_time": "03:00",  # UTC time
    "paused": False
}

connector_id = "connector_abc123"
response = requests.patch(
    f"{base_url}/connectors/{connector_id}",
    auth=(api_key, api_secret),
    headers={"Content-Type": "application/json"},
    data=json.dumps(sync_config)
)

This skill helps you move from manual UI configuration to repeatable, version-controlled infrastructure that scales across environments and teams.