Generate Advanced Buildkite CI/CD Pipelines

This skill is a Buildkite pipeline expert covering task orchestration, conditional execution, and scalable architectural patterns - Buildkite's agent model, dynamic pipeline generation, block steps, matrix builds, and artifact management. Core principles: agent-queue strategy (target agent selectors by workload - compute-intensive, platform-specific, specialized tooling); step dependencies via depends_on and wait steps for efficient parallelism with required sequencing; artifact flow via artifact_paths and plugins; and environment isolation via scoped environment variables and secrets through hooks/plugins.

Use it when a CI/CD pipeline needs proper task orchestration on Buildkite specifically - agent targeting, parallel test steps, artifact handoff between steps - rather than a flat, unstructured pipeline.

steps:
  - label: ":hammer: Build"
    command: |
      echo "Building application..."
      make build
    artifact_paths:
      - "dist/**/*"
      - "build-info.json"
    agents:
      queue: "build"
      os: "linux"
    timeout_in_minutes: 15

  - wait: ~
    continue_on_failure: false

  - label: ":test_tube: Test Suite"
    command: "make test"
    parallelism: 3
    agents:
      queue: "test"

Beyond the basic structure, it covers dynamic pipeline generation (a Python script that inspects changed services and emits per-service build/test steps, uploaded via buildkite-agent pipeline upload), matrix builds (testing across a node/os matrix with ${matrix.*} variable interpolation), block steps for manual approval gates (a deploy prompt with a select field for environment and a text field for release notes, consumed downstream via BUILDKITE_BUILD_META_DATA_*), plugin integration (Docker builds with pinned plugin versions, Slack notifications gated on build.state == "passed"), conditional execution by branch (branches: "!main !production") or if expression (checking build.branch, build.env(...), or build.message patterns), and error handling (automatic retry rules keyed by exit status with a retry limit, plus soft_fail and continue_on_failure for non-blocking steps).

Named plugins include docker, artifacts, slack, and junit-annotate for test-result reporting, each pinned to a specific version in the examples.

Platform and DevOps engineers building or hardening a Buildkite pipeline who need concrete, working patterns for dependencies, matrix testing, manual approval gates, and retry/failure handling rather than assembling them from Buildkite's documentation piece by piece. Five performance and scalability tips round out the guidance: use parallelism for independent build/test steps, target specific agent queues to optimize resource use, minimize artifact size with selective downloading, apply proper build caching through plugins or agent-level caching, and balance step granularity so pipelines avoid both excessive per-step overhead and poorly-parallelized monolithic steps. The overarching design goal stated throughout is to fail fast and give clear feedback via step labels and artifacts, while keeping an eye on both developer experience and infrastructure cost.