Master Workflow Orchestration Patterns

This skill covers workflow orchestration architecture with Temporal, centered on the fundamental design decision of what belongs in a Workflow (orchestration and decision-making, which must be deterministic and cannot make direct external calls) versus an Activity (all external interactions - API calls, database writes - which can be non-deterministic but must be idempotent since Temporal may execute them more than once). It gives a one-line decision framework: touches external systems -> Activity; is orchestration/decision logic -> Workflow.

It documents four core orchestration patterns: the Saga pattern with compensation for distributed transactions (register each step's compensation before executing it, then run all compensations in reverse/LIFO order on failure - illustrated with a payment workflow reserving inventory, charging payment, and fulfilling an order, each with its own rollback); Entity Workflows (the actor model, where one long-lived workflow execution represents a single entity instance - a shopping cart, bank account, or inventory item - receiving signals for state changes and supporting queries for current state); Fan-Out/Fan-In for parallel execution (spawning child workflows or parallel activities and aggregating results, with an explicit scaling rule: don't scale individual workflows arbitrarily large - for 1M tasks, spawn 1K child workflows of 1K tasks each rather than one workflow trying to handle everything); and the Async Callback pattern for waiting on external events or human approval via signals.

It covers hard determinism constraints workflows must obey (no threading/locks, no random(), no global/static state, no system time via datetime.now(), no direct I/O or network calls, no non-deterministic libraries - only workflow.now(), workflow.random(), pure functions, and activity calls are allowed), versioning strategies for changing workflow code while old executions are still running (workflow.get_version(), routing new executions to a new workflow type, ensuring old events still replay correctly), retry policy configuration (backoff coefficient, max interval, max attempts, and explicitly non-retryable error classes like validation failures), idempotency implementation strategies (dedup keys, check-then-act with unique constraints, upserts instead of inserts), and activity heartbeats for detecting stalled long-running activities.

Use Temporal workflow orchestration for multi-step processes spanning machines/services/databases, distributed transactions needing all-or-nothing semantics, long-running workflows (hours to years) with automatic state persistence, failure recovery that must resume from the last successful step, business processes like bookings/orders/approvals, entity lifecycle management, infrastructure automation (CI/CD, provisioning), or human-in-the-loop systems needing timeouts and escalation.

Do not use it for simple CRUD operations (direct API calls suffice), pure data processing pipelines (use Airflow or batch processing instead), stateless request/response (standard APIs), or real-time streaming (use Kafka or event processors instead) - these are explicitly named as poor fits for a workflow engine.

Input is a distributed-systems reliability problem - a multi-step business process, a transaction needing compensation on failure, an entity needing lifecycle tracking, or parallel work needing aggregation. Output is a workflow/activity design applying the appropriate pattern (Saga, Entity, Fan-Out/Fan-In, Async Callback), with correct determinism boundaries, retry/idempotency configuration, and a versioning strategy for safe future changes.

Built specifically around Temporal's execution model (workflows, activities, event history, signals, queries, child workflows), citing Temporal's own documentation and blog posts (docs.temporal.io, temporal.io/blog) as the authoritative source for each pattern and principle.

Architects and engineers designing distributed workflow systems on Temporal who need to get the workflow-vs-activity boundary and core orchestration patterns right from the start, rather than debugging determinism violations or non-idempotent retries after the fact.