Spark architecture

A driver plans the work; executors run one task per partition

Driveryour program · builds the plansplits it into stages and tasksExecutor 1task · core 1task · core 2memory · local diskExecutor 2task · core 1task · core 2memory · local diskExecutor 3task · core 1task · core 2memory · local diskExecutor 4task · core 1task · core 2memory · local diskThis query’s first stage: the file splits into 3 partitions, so 3 tasks run in parallel and 5 cores wait.Spark splits data into partitions and work into tasks, one task per partition, run in parallel across executors.The driver turns your code into a plan, cuts it into stages, and schedules one task per partition on the executors’ cores. A cluster of fourexecutors with two cores runs eight tasks at once; more partitions than cores queue up, fewer leave cores idle. Measured: the 1 million ordersfile was read as 3 partitions. Partitions are the unit of parallelism, so their number and their sizes decide how fast a job runs.

The driver plans; executors run tasks, one per partition, in parallel on their cores.

Spark architecture · Going deeper · 1 of 2

DataFrames, SQL and where your code runs

The DataFrame API and Spark SQL compile to the same plan; Python UDFs run outside it and are slower.

Two ways to write the same job

  • DataFrame API df.filter(col("status") != "cancelled").groupBy("m").agg(sum("total")) in Python, Scala, R or Java.
  • Spark SQL SELECT month(order_date) m, sum(total) FROM orders WHERE status <> 'cancelled' GROUP BY m. Both go through the same optimiser (Catalyst) and run the same physical plan.
  • RDDs the original low-level API of objects and functions. The optimiser cannot see inside them; use DataFrames unless you need full control.

Things that slow PySpark down

Python UDFseach row is sent from the executor’s JVM to a Python process and back. Built-in functions, or vectorised pandas UDFs, avoid most of this.
collect()brings every row to the driver. Fine for 12 monthly totals, fatal for a billion rows: write results to a table instead.
Tiny filesmillions of small files mean millions of tiny tasks and slow listing. Compact them (OPTIMIZE).

repartition and coalesce · Going deeper · 2 of 2

repartition shuffles to even sizes; coalesce only merges

start: 4 partitions250,000 rows eachdf.repartition(8)8 × 125,000: exactly evenExchange RoundRobinPartitioning(8)a full shuffle: every row moves.coalesce(2)2 × 500,000Coalesce 2no Exchange: no shuffledf.repartition(8, "customer")123,163 to 126,057 rowshashpartitioning(customer, 8)one customer, one partition✗coalesce(16) on 8 partitions still gives 8: it can only reduce.✓One output file per partition: the 4-partition write made 4 Parquet files; after coalesce(1), 1 file.Measured on PySpark 3.5.5, 1,000,000 rows, local[4]. Bars are rows per partition, to scale.repartition shuffles every row into even partitions; coalesce merges without a shuffle, and only reduces.Use coalesce to cut the number of output files cheaply. Use repartition when partitions are uneven, when you need more ofthem, or to put each key in one partition before a per-key step. A coalesce(1) at the end is not free: with no shuffle toseparate it, the work before it also runs as one task. On Delta tables, OPTIMIZE compacts small files afterwards.df.write.partitionBy("order_date") is a different thing: it lays the files out in one folder per value, which is what lets aquery prune by date.

repartition(n) does a full shuffle and can raise or lower the count; coalesce(n) merges partitions with no shuffle.

Lazy evaluation

Transformations only describe the result; an action runs the plan

df = spark.read.parquet('orders.parquet')q = (df.filter(col('status') != 'cancelled') .withColumn('m', month('order_date')) .groupBy('m').agg(sum('total')))transformationsnothing is read yet: Spark onlyrecords what you asked forq.collect() # an action: now a job runsactions: collect, count, show, writeThe optimised plan, from explain():Scan parquet PushedFilters: [Not(EqualTo(status,cancelled))] ReadSchema: struct<order_date, status, total>Transformations only describe a result; an action makes Spark optimise the whole plan, then run it.Because nothing runs until an action, the optimiser sees the full query first. Measured: it pushed the status filter into the Parquet readerand read only 3 of the 10 columns, so skipped data never left the disk. The flip side: an error in a transformation appears only when an actionruns, and each action re-computes from the source unless you cache the DataFrame.

Transformations such as filter and groupBy are lazy; an action such as collect or write triggers the optimised job.

cache and persist · Going deeper

cache() marks a result to keep; the next action fills it

agg = df.groupBy("customer").sum("total")agg.cache() # lazy: stores nothing yetagg.count() # runs the job, fills the cacheagg.count() # read back from the cacheagg.unpersist() # give the memory backThe plan of the second count, from explain():InMemoryTableScan [customer, sum(total)]+- InMemoryRelation StorageLevel(disk, memory, deserialized, 1 replicas) +- HashAggregate … Exchange … RangeEverything under InMemoryRelation ran once; later actions start from the stored result.Storage levelsMEMORY_AND_DISKcache() on a DataFrame: the overflow goes to local diskMEMORY_ONLYcache() on an RDD: the overflow is recomputedDISK_ONLYslower to read, uses no executor memoryMEMORY_AND_DISK_2a level ending _2 is kept on two executorsdf.persist(StorageLevel.DISK_ONLY)cache() only marks a DataFrame; the next action fills the cache, and later actions read from it.Cache a result that is expensive and used by two or more actions: a cleaned table feeding several aggregates, or a model’straining set. Do not cache what is read once: filling the cache costs time, and cached data takes memory the executors needfor shuffles. Call unpersist() when done. Databricks’ disk cache is separate: it keeps copies of remote Parquet files onlocal SSD automatically. Storage level printed by PySpark 3.5.5 as “Disk Memory Deserialized 1x Replicated”.

cache() stores a DataFrame at MEMORY_AND_DISK after the next action; persist() takes another storage level.

Shuffles and stages

Grouping and joining move rows between machines, splitting stages

Stage 1: narrow (no data moves)Stage 2: after the shufflepartition 1filter, partial sumpartition 2filter, partial sumpartition 3filter, partial sumshuffle partition 1shuffle partition 2shuffle partition 3rows hashed by month:each month lands inone place: final sum15 partial rows in all200 shuffle partitions planned; AQE coalesced themA wide transformation such as groupBy or join must move rows between machines; that shuffle splits the job into stages.Narrow steps (filter, select, withColumn) work on each partition alone, chained in one stage. A groupBy needs every row of a monthtogether, so rows are hashed by key and sent across the network: the shuffle, the costliest step in Spark. Measured: Spark summed within eachpartition first, so only 15 partial rows (720 bytes) crossed the shuffle instead of 1,000,000 orders. Exchange in a plan marks a shuffle.

Narrow transformations stay within a partition; wide ones such as groupBy need a shuffle, which starts a new stage.

Spark settings · Going deeper

A handful of settings, and their defaults, cut up every job

SettingDefaultWhat it decidesspark.sql.shuffle.partitions200partitions after every shuffle, whatever the data sizespark.sql.files.maxPartitionBytes128 MBhow large each input partition is when files are readspark.sql.autoBroadcastJoinThreshold10 MBthe largest table broadcast without a hint; -1 turns it offspark.sql.adaptive.advisoryPartitionSizeInBytes64 MBthe size AQE aims for when it merges small shuffle partitionsspark.sql.adaptive.skewJoin.skewedPartitionFactor5AQE calls a partition skewed above 5× the median and 256 MBspark.sql.sources.partitionOverwriteModeSTATICoverwrite replaces every partition; DYNAMIC only those writtenspark.executor.memory1gheap per executor; too small shows as spill, then out-of-memoryspark.memory.fraction0.6share of that heap for execution and cached data togetherDefaults as read from spark.conf on Spark 3.5.5. Set per session with spark.conf.set(name, value), or with --conf at spark-submit.The defaults suit a few gigabytes: 200 shuffle partitions, 128 MB input splits, broadcasts up to 10 MB.A 500 GB shuffle cut into 200 partitions makes each 2.5 GB, which spills to disk; at a rule-of-thumb 128 MB each it needsabout 4,000. AQE merges partitions that are too small but does not split ones that are too big, except skewed ones in joins,so err on the high side. Databricks accepts spark.sql.shuffle.partitions = auto. STATIC overwrite is the trap: writing oneday with mode("overwrite") into a partitioned table deletes every other day, unless the mode is DYNAMIC or you usereplaceWhere.

Defaults: spark.sql.shuffle.partitions is 200, input partitions are 128 MB, and tables under 10 MB are broadcast.

Broadcast joins

Copy a small table to every executor and the large one never moves

Sort-merge join: both sides shuffledorders1,000,000 rowscustomers50,000 rowsExchange (shuffle)Exchange (shuffle)SortMergeJoinBroadcast join: small side copied to every executorordersstays where it iscustomerssent whole to each executorBroadcastExchangeBroadcastHashJoinboth plans measured; left: broadcasting switched offWhen one side of a join is small, Spark broadcasts it to every executor so the large side never moves.A sort-merge join shuffles both tables by the join key so matching rows meet: 1,000,000 orders cross the network. A broadcast join copiesthe small table to every executor, and each partition of orders joins locally. Spark broadcasts automatically under 10 MB(spark.sql.autoBroadcastJoinThreshold); a hint, broadcast(df), forces it. Joins of two large tables still need the shuffle.

A broadcast join sends the small table to every executor, avoiding a shuffle; Spark does it automatically under 10 MB.

Data skew

One oversized partition makes the whole stage wait

Rows per partition, keyed by customerp0124kp1125kp2126kp3125kp4127kp5125kp6124kp7124k… when 40% of orders share one customer idp074kp175kp276kp3475kp476kp575kp674kp775kthe stage waits for p3: 6.3× the median taskSkew is one partition holding far more data than the rest; the whole stage waits for that one slow task.Measured: hashing a million orders by customer into 8 partitions gave 123,534 to 126,977 rows each. Give 40% of orders one customer (aguest checkout id, a null, a test account) and that key’s partition holds 475,017 rows while the others hold about 75,056. Fixes: let AQE splitskewed join partitions, filter or handle the hot key separately, broadcast the small side, or salt the key (add a random suffix, aggregate twice).

Data skew puts far more rows in one partition: here 475,017 against about 75,056, so one task runs long.

Data skew · Going deeper

Salting a skewed key

Salting spreads one hot key over several partitions by adding a random suffix, then combines the partial results.

Two-step aggregation

  • 1. Add a salt withColumn("salt", (rand() * 16).cast("int")). The hot customer’s rows now carry 16 different (customer, salt) pairs.
  • 2. Aggregate by key and salt groupBy("customer_id", "salt").agg(sum("total")): the hot key is spread over up to 16 tasks.
  • 3. Aggregate again by key groupBy("customer_id").agg(sum("sum(total)")): 16 small rows per hot key, cheap to combine.
For joins, salt the large side randomly and replicate each row of the small side once per salt value. Try AQE’s skew join handling first: it splits oversized shuffle partitions automatically when spark.sql.adaptive.skewJoin.enabled is on (the default).

Spill

What it isa task’s partition does not fit in its share of executor memory, so Spark writes intermediate data to local disk and reads it back.
Where to see itSpark UI, stage page: “Spill (Memory)” and “Spill (Disk)” columns.
Fixesmore, smaller partitions; remove skew; executors with more memory per core.

The Spark UI

A max far above the median means skew; spill means memory ran out

Spark UI · Stage · Summary metrics for 8 completed tasksMetricMin25thMedian75thMaxRecords read74,25274,57775,18175,554475,017Duration…………longestShuffle read size…………largestSpill (disk)0000if it did not fitMax 6.3× the median: skew. Look first at the max column, then at which stage it is in.Records read are this lesson’s measured skewed partitions; the other rows show what to look for.The Spark UI shows each stage’s tasks; a maximum far above the median means skew, and spill means memory ran out.Jobs are split into stages at each shuffle; the stage page lists task metrics as min, quartiles and max. Even tasks with a long total time point totoo little parallelism or too much data read; one huge maximum points to skew; non-zero spill means partitions are too big for memory; largeshuffle read and write point to a join or groupBy that could broadcast, or a filter that should come earlier. On Databricks the query profileshows the same for SQL.

In the Spark UI, compare each stage’s task max with its median, and look for spill and large shuffle sizes.

Adaptive Query Execution

AQE re-plans the query at each shuffle using real sizes

Adaptive Query Execution re-plans at each shuffle, using the sizes it has just measuredCoalesce partitions200 planned, mostly tinymerged into a few full onesSwitch join strategyplanned: SortMergeJoinafter a filter, one sideturns out to be 4 MBruns: BroadcastHashJoinSplit skewed partitionsone partition 6× the mediansplit across several tasksAdaptive Query Execution fixes the plan while the query runs, once real data sizes are known.It is on by default (Spark 3.2 and later, and on Databricks). Measured: every plan in this lesson shows AQEShuffleRead coalesced, merging the 200default shuffle partitions into a few. It also turns a sort-merge join into a broadcast join when one side proves small, and splits oversizedpartitions in skewed joins. It cannot fix skew in a plain groupBy, a missing filter, or a Python UDF; those are still yours to find in the UI.

Adaptive Query Execution coalesces small shuffle partitions, switches to broadcast joins and splits skewed partitions at runtime.

Databricks compute

Choose compute by who is waiting; let the table layout cut reading

ComputeUse it forNotesAll-purpose clusterinteractive notebooks, developmentshared by users; stop when idleJobs computescheduled production jobscreated for the run, then terminated; cheaperServerlessnotebooks, jobs, pipelinesstarts in seconds, no cluster to configureSQL warehouseSQL, dashboards, BI toolsPhoton engine; serverless or classicTable layoutLiquid clustering: CLUSTER BY (customer_id, order_date) keeps related rows in the same files and can change keys later.Predictive optimization: Unity Catalog runs OPTIMIZE, VACUUM and ANALYZE on managed tables automatically.On Databricks, pick compute by who is waiting, and let the table layout keep queries reading as little as possible.Run production work on jobs compute or serverless, not on an always-on interactive cluster. Photon, Databricks’ vectorised engine, speeds upSQL and DataFrame work. Liquid clustering replaces hand-picked partition columns and Z-ordering for new tables, and predictive optimizationremoves the chore of compacting and vacuuming. Most slow jobs are still fixed by reading less: filter early, select fewer columns, avoid skew.

Use jobs compute or serverless for production, SQL warehouses for BI; liquid clustering and predictive optimization keep tables fast.

Lesson complete

You finished “Batch processing with Spark”

Nicely done.

Open in a new tab