Generate Production-Ready Flink Data Pipelines

This skill generates production-ready Apache Flink streaming and batch jobs across the DataStream API, Table API, CEP, state management, checkpointing, and deployment. Every job architecture covers six components: environment setup (parallelism and checkpointing configuration), source configuration (reliable sources with proper serialization), processing logic (transformations, windowing, aggregations), sink configuration (fault-tolerant output), state management (proper handling and recovery), and monitoring integration (metrics and logging). A typical DataStream job sets parallelism, enables exactly-once checkpointing on a fixed interval with a minimum pause between checkpoints, configures a HashMap state backend with S3 checkpoint storage, consumes from Kafka with named and uid'd operators, applies a keyBy plus tumbling event-time window and aggregation, and produces to Kafka with exactly-once semantics.

Use it when building a Flink job that needs real production concerns handled - checkpointing, state recovery, watermarking for out-of-order data, and Kubernetes deployment - not a toy word-count example. It is not a shortcut around its own best practices: operators should always have explicit UIDs for state compatibility across upgrades, and jobs should be tested with realistic data volumes and failure scenarios before production.

Given a streaming or batch requirement, it produces a keyed-state processor pattern (a KeyedProcessFunction tracking a per-key event count and last-seen timestamp, firing an alert and clearing state past a threshold, and registering a 24-hour event-time timer for cleanup), a Table API job (a Kafka-backed source table with a bounded-out-of-orderness watermark, a JDBC sink table, and an hourly tumbling-window aggregation query filtered to a specific event type), resource and checkpoint tuning (task manager memory sizing, task slot count, a 5-minute checkpoint timeout, a cap of one concurrent checkpoint, externalized checkpoint retention on cancellation, and network buffer memory fractions), a custom watermark strategy (10-second bounded out-of-orderness with a 1-minute idleness timeout), and custom metrics (a Counter for events processed and a Histogram for processing time, both registered through the runtime metric group).

apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  name: flink-streaming-job
spec:
  image: flink:1.17.1-scala_2.12-java11
  flinkVersion: v1_17
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "2"
    state.backend: hashmap
    state.checkpoints.dir: s3://checkpoints/
    execution.checkpointing.interval: 60s
  serviceAccount: flink
  jobManager:
    resource:
      memory: "2048m"
      cpu: 1
  taskManager:
    resource:
      memory: "4096m"
      cpu: 2
  job:
    jarURI: s3://jars/flink-streaming-job.jar
    parallelism: 4
    upgradeMode: stateless

Sources and sinks connect through Kafka (with exactly-once producer semantics), JDBC (Postgres in the Table API example), and S3 for checkpoint storage, deployed via the Flink Kubernetes Operator's FlinkDeployment custom resource with separate job-manager and task-manager resource requests.

Data engineers building production Flink pipelines who need the operational details, not just the processing logic - always-set operator UIDs for safe upgrades, parallelism sized to data volume and available resources, event-time processing with proper watermark strategies, side-output error handling for malformed records, backpressure monitoring with tuned buffer sizes, async I/O for external lookups, and data partitioning that avoids skew and hotspots.