Build & Optimize Google Cloud Dataflow Templates

Google Cloud Dataflow Template Expert Agent builds classic and Flex Dataflow templates using Apache Beam: parameterized pipelines with ValueProvider runtime parameters, nested JSON-backed configuration classes, dead-letter-queue error handling, windowing and triggering for streaming data, state and timer-based streaming logic, resource tuning for machine type and JVM/runner settings, unit and integration testing, and staged deployment with monitoring labels.

Use it when building or troubleshooting a Dataflow pipeline - choosing between a classic template (simple, parameter-driven) and a Flex template (custom dependencies via a Docker image), adding a dead-letter queue for failed records, windowing streaming data with allowed lateness, using state and timers for buffering logic, tuning worker machine type and JVM settings, or staging and deploying a versioned template. It's Apache Beam and Dataflow-specific - the Java SDK patterns, ValueProvider parameters, and gcloud dataflow commands don't transfer to other data-processing frameworks like Spark or Flink.

Input is the pipeline's source, transform, and sink, plus whether it runs as batch or streaming. Output is Java Beam code and deployment configuration: a classic @Template-annotated BigQueryToPubSubTemplate reading a BigQuery query and publishing rows to Pub/Sub; a Flex-template Dockerfile setting the Java main class and classpath; ValueProvider-typed pipeline options for input subscription, window duration, and a dead-letter topic; a TransformConfig class deserializing nested JSON configuration via Gson; a ProcessWithErrorHandling DoFn routing failures to a dead-letter output with error metadata; a fixed-window transform with watermark-based triggering, early firings, and allowed lateness; a stateful DoFn using BagState and a processing-time timer to buffer and flush elements; a template metadata JSON declaring a machineType parameter; a gcloud dataflow flex-template build command with runner v2 and portable job submission experiments; a TestPipeline-based unit test asserting transform output; and a gcloud dataflow flex-template run deployment command with region, parameters, and monitoring labels.

Built on the Apache Beam Java SDK targeting Google Cloud Dataflow, with BigQueryIO and PubsubIO connectors, Docker for Flex template images, and gcloud/gsutil for staging and deployment:

@ProcessElement
public void processElement(@Element String element, 
                         OutputReceiver<String> mainOutput,
                         OutputReceiver<String> deadLetterOutput,
                         ProcessContext context) {
    try {
        // Process element
        String result = processData(element);
        mainOutput.output(result);
    } catch (Exception e) {
        // Send to dead letter queue with error metadata
        String errorRecord = createErrorRecord(element, e);
        deadLetterOutput.output(errorRecord);
    }
}

Data engineers building or operating Dataflow pipelines who want production-ready patterns - dead-letter handling, tested windowing and state logic, tuned worker resources, staged and versioned deployment, structured logging with correlation IDs, alerting on job failures, and least-privilege IAM - rather than a bare pipeline with no error handling or observability.