Optimize Apache Spark Performance

This skill provides production patterns for optimizing Apache Spark jobs, including partitioning strategies, memory management, shuffle optimization, and performance tuning. The execution model flows from a driver program through jobs (triggered by an action) into stages (separated by shuffles) into tasks (one per partition). Key performance factors and their fixes: shuffle (network/disk I/O, minimize wide transformations), data skew (uneven task duration, fixed via salting or broadcast joins), serialization (CPU overhead, use Kryo and columnar formats), memory (GC pressure and spills, tune executor memory), and partitions (parallelism, right-size them).

Use this skill when optimizing slow Spark jobs, tuning memory/executor configuration, implementing efficient partitioning, debugging performance issues, scaling pipelines for large datasets, or reducing shuffle and data skew. A quick-start config enables Adaptive Query Execution (spark.sql.adaptive.enabled, coalescePartitions.enabled, skewJoin.enabled), the Kryo serializer, and 200 shuffle partitions, then reads Parquet and runs a filter/select/groupBy/agg pipeline.

Seven patterns cover the core techniques. Optimal partitioning targets 128-256MB per partition, using repartition() for even distribution, coalesce() to reduce partitions without a shuffle, predicate pushdown on filtered reads, and partitionBy() on writes for future query pruning. Join optimization covers broadcast joins for small tables (under ~10MB via F.broadcast()), sort-merge joins as the shuffle-based default for large tables, bucketed tables (pre-sorted, pre-bucketed on the same key/count to avoid a shuffle at join time), and skew handling via AQE's skew-join settings (skewedPartitionFactor, skewedPartitionThresholdInBytes) or manual salting (adding a random salt column to the skewed side, exploding the other side across all salt values, then joining on the salted key). Caching and persistence covers .cache()/.persist() with explicit storage levels (MEMORY_ONLY, MEMORY_AND_DISK recommended, MEMORY_ONLY_SER, DISK_ONLY, OFF_HEAP), forcing materialization with .count(), reusing a cached DataFrame across multiple aggregations, unpersisting when done, and checkpointing to break complex lineage.

Memory tuning breaks down an 8GB executor's memory (60% via spark.memory.fraction for execution+storage, 50% of that via spark.memory.storageFraction for cache, the rest for execution; the remaining 40% for user data structures), alongside memoryOverhead for non-JVM memory, autoBroadcastJoinThreshold, and maxPartitionBytes to prevent OOM on large shuffles; executor memory status can be queried programmatically per executor. Shuffle optimization pre-aggregates locally before a global aggregation, uses approx_count_distinct instead of an exact distinct().count() to avoid a shuffle, prefers coalesce() over repartition() when only reducing partition count, and enables shuffle/spill compression (lz4). Data format optimization covers Parquet write options (Snappy compression, 128MB row groups), column pruning, predicate pushdown, and Delta Lake's optimizeWrite/autoCompact plus OPTIMIZE ... ZORDER BY for multi-dimensional query performance. Monitoring/debugging covers df.explain() in various modes (simple, extended, codegen, cost, formatted), programmatically inspecting stage task counts via statusTracker(), and detecting partition skew by counting rows per spark_partition_id() and computing a max/avg skew ratio (over 2x indicates skew). A production configuration cheat sheet bundles AQE, memory, parallelism, serialization, compression, broadcast, and file-handling settings together. Best practices: enable AQE, use Parquet/Delta, broadcast small tables, monitor the Spark UI for skew/spills/GC, and right-size partitions; avoid collecting large data to the driver, unnecessary UDFs, over-caching, ignoring skew, and using .count() just to check existence (prefer .take(1)/.isEmpty()).

Data engineers tuning slow or resource-heavy Spark jobs who need concrete partitioning, join, memory, shuffle, and file-format optimization patterns plus debugging techniques for skew and stage-level metrics.