Build & Optimize Google Cloud Dataflow Templates

The Google Cloud Dataflow Template Expert Agent covers building reusable Dataflow templates on the Apache Beam SDK for both batch and streaming workloads. It distinguishes Classic Templates - simple, parameter-driven Java pipelines using a @Template annotation and PipelineOptions interfaces with ValueProvider fields - from Flex Templates, which package custom dependencies into a Docker image built from the dataflow-templates-base launcher base image with FLEX_TEMPLATE_JAVA_MAIN_CLASS and FLEX_TEMPLATE_JAVA_CLASSPATH environment variables. On pipeline parameters it shows using ValueProvider for every runtime parameter (input subscription, window duration with a default, dead-letter topic) and structuring complex configuration as nested, JSON-deserializable parameter classes. It covers robust error handling via a DoFn that routes failures to a dead-letter output with error metadata instead of failing the pipeline. For streaming it demonstrates fixed windowing with watermark-based triggers, early firings, allowed lateness, and accumulating fired panes, plus stateful processing using BagState and a processing-time Timer to buffer and flush elements. On resource optimization it shows a template metadata JSON defining a configurable machineType parameter, and building a Flex template with gcloud dataflow flex-template build using the use_runner_v2 and use_portable_job_submission experiments. For testing it shows a TestPipeline/PAssert unit test asserting exact output content, and notes integration testing should validate realistic data volumes, error scenarios, and data quality. For deployment it covers staging a versioned template to a GCS bucket and running it with labeled, timestamped job names.

Use it when authoring a new Dataflow template (Classic or Flex), adding ValueProvider-based runtime parameters, implementing dead-letter error handling, designing windowing/triggering for a streaming pipeline, tuning worker machine type or JVM settings, writing unit tests with TestPipeline, or staging and deploying a template with gcloud dataflow flex-template. It is not for ad-hoc one-off Beam pipelines that will never be templated or reused, and it is not a substitute for validating IAM roles and least-privilege access separately - the agent notes proper IAM and structured logging with correlation IDs as deployment requirements but does not configure them for you.

Inputs are pipeline source code (Java/Apache Beam), a template metadata JSON describing parameters and machine type, and for Flex Templates a Docker image built from the launcher base image. Outputs are a staged, versioned Dataflow template in a GCS bucket and running Dataflow jobs launched from that template with runtime parameters and monitoring labels.

Apache Beam SDK (Java), BigQuery and Pub/Sub as example sources/sinks (BigQueryIO, PubsubIO), Docker for Flex Template packaging, and the gcloud CLI for building and running templates:

gcloud dataflow flex-template run "job-$(date +%Y%m%d-%H%M%S)" \
  --template-file-gcs-location gs://bucket/template \
  --region us-central1 \
  --parameters inputTopic=projects/project/topics/input \
  --labels env=prod,version=v1-2-3

Data engineers building reusable, production-grade Dataflow templates who need concrete Apache Beam patterns for parameterization, error handling, windowing, and deployment rather than a general Beam programming-model overview.