Spark architecture
A driver plans the work; executors run one task per partition
The driver plans; executors run tasks, one per partition, in parallel on their cores.
Spark processes data too big for one machine by splitting it. The driver runs your program. It turns your code into a plan, cuts the plan into stages and each stage into tasks. Executors on the worker machines run those tasks, one per core at a time, each on one partition of the data. Four executors with two cores run eight tasks at once. When we read our million-order file, it came in as 3 partitions, so 3 tasks ran and five cores waited. Partitions are the unit of parallelism.
Spark architecture · Going deeper · 1 of 2
DataFrames, SQL and where your code runs
The DataFrame API and Spark SQL compile to the same plan; Python UDFs run outside it and are slower.
The DataFrame API and Spark SQL compile to the same plan; Python UDFs run outside it and are slower. Two ways to write the same job. DataFrame API df.filter(col("status") != "cancelled").groupBy("m").agg(sum("total")) in Python, Scala, R or Java. Spark SQL SELECT month(order_date) m, sum(total) FROM orders WHERE status <> 'cancelled' GROUP BY m. Both go through the same optimiser (Catalyst) and run the same physical plan. RDDs the original low-level API of objects and functions. The optimiser cannot see inside them; use DataFrames unless you need full control. Things that slow PySpark down.
Two ways to write the same job
- DataFrame API df.filter(col("status") != "cancelled").groupBy("m").agg(sum("total")) in Python, Scala, R or Java.
- Spark SQL SELECT month(order_date) m, sum(total) FROM orders WHERE status <> 'cancelled' GROUP BY m. Both go through the same optimiser (Catalyst) and run the same physical plan.
- RDDs the original low-level API of objects and functions. The optimiser cannot see inside them; use DataFrames unless you need full control.
Things that slow PySpark down
repartition and coalesce · Going deeper · 2 of 2
repartition shuffles to even sizes; coalesce only merges
repartition(n) does a full shuffle and can raise or lower the count; coalesce(n) merges partitions with no shuffle.
Lazy evaluation
Transformations only describe the result; an action runs the plan
Transformations such as filter and groupBy are lazy; an action such as collect or write triggers the optimised job.
Spark is lazy, on purpose. Reading, filtering, adding a column, grouping: none of these touch the data. Spark only records what you asked for. Then an action, here collect, asks for an actual result, and a job runs. Because Spark saw the whole query first, the optimiser could improve it. In the real plan, the status filter was pushed into the Parquet reader, and only three of the ten columns were read.
cache and persist · Going deeper
cache() marks a result to keep; the next action fills it
cache() stores a DataFrame at MEMORY_AND_DISK after the next action; persist() takes another storage level.
Shuffles and stages
Grouping and joining move rows between machines, splitting stages
Narrow transformations stay within a partition; wide ones such as groupBy need a shuffle, which starts a new stage.
Some steps can be done partition by partition. Filtering and adding a column need nothing from other partitions, so they run together in one stage. Spark even sums within each partition first. But a total per month needs every row for that month in one place. So rows are hashed by month and sent across the network: the shuffle. After it, a new stage finishes the sums. Here only 15 partial rows crossed the network, 720 bytes, because of that early summing. Shuffles are where most Spark time goes.
Spark settings · Going deeper
A handful of settings, and their defaults, cut up every job
Defaults: spark.sql.shuffle.partitions is 200, input partitions are 128 MB, and tables under 10 MB are broadcast.
Broadcast joins
Copy a small table to every executor and the large one never moves
A broadcast join sends the small table to every executor, avoiding a shuffle; Spark does it automatically under 10 MB.
Joins are wide too, usually. A sort-merge join shuffles both tables by the join key so matching rows meet: all million orders cross the network. But the customer table is small. A broadcast join sends a full copy of it to every executor, and each partition of orders joins against it where it already sits. No shuffle of the big table at all. Spark chooses this by itself when a side is under 10 megabytes, and you can force it with a broadcast hint.
Data skew
One oversized partition makes the whole stage wait
Data skew puts far more rows in one partition: here 475,017 against about 75,056, so one task runs long.
A stage is only as fast as its slowest task. Hash a million orders by customer into eight partitions and they come out even, about a hundred and twenty-five thousand each. Now give forty percent of the orders the same customer id, as guest checkouts or nulls often do. That key's partition holds 475,017 rows, the others about 75,056. Seven tasks finish and wait for one. The fixes: let adaptive execution split it, treat the hot key separately, broadcast, or salt the key.
Data skew · Going deeper
Salting a skewed key
Salting spreads one hot key over several partitions by adding a random suffix, then combines the partial results.
Salting spreads one hot key over several partitions by adding a random suffix, then combines the partial results. Two-step aggregation. 1. Add a salt withColumn("salt", (rand() * 16).cast("int")). The hot customer’s rows now carry 16 different (customer, salt) pairs. 2. Aggregate by key and salt groupBy("customer_id", "salt").agg(sum("total")): the hot key is spread over up to 16 tasks. 3. Aggregate again by key groupBy("customer_id").agg(sum("sum(total)")): 16 small rows per hot key, cheap to combine. For joins, salt the large side randomly and replicate each row of the small side once per salt value. Try AQE’s skew join handling first: it splits oversized shuffle partitions automatically when spark.sql.adaptive.skewJoin.enabled is on (the default). Spill.
Two-step aggregation
- 1. Add a salt withColumn("salt", (rand() * 16).cast("int")). The hot customer’s rows now carry 16 different (customer, salt) pairs.
- 2. Aggregate by key and salt groupBy("customer_id", "salt").agg(sum("total")): the hot key is spread over up to 16 tasks.
- 3. Aggregate again by key groupBy("customer_id").agg(sum("sum(total)")): 16 small rows per hot key, cheap to combine.
Spill
The Spark UI
A max far above the median means skew; spill means memory ran out
In the Spark UI, compare each stage’s task max with its median, and look for spill and large shuffle sizes.
When a job is slow, open the Spark UI and find the slowest stage. Its summary table lists each metric as minimum, quartiles and maximum across tasks. Here records read go from 74,252 to 475,017. A maximum that many times the median is skew. Spill to disk means a task's data did not fit in memory. Big shuffle reads point to a join or grouping you could avoid or broadcast.
Adaptive Query Execution
AQE re-plans the query at each shuffle using real sizes
Adaptive Query Execution coalesces small shuffle partitions, switches to broadcast joins and splits skewed partitions at runtime.
The optimiser plans before it has seen the data. Adaptive Query Execution corrects the plan at each shuffle, when it knows the real sizes. The default 200 shuffle partitions are often tiny; it merges them into a few full ones. Every plan in this lesson shows that. If a join side turns out small after filtering, it switches to a broadcast join. And if one partition of a join is far larger than the rest, it splits it across several tasks.
Databricks compute
Choose compute by who is waiting; let the table layout cut reading
Use jobs compute or serverless for production, SQL warehouses for BI; liquid clustering and predictive optimization keep tables fast.
On Databricks, the compute you choose matters for cost. Interactive all-purpose clusters are for development. Scheduled jobs belong on jobs compute, created for the run and then terminated, or on serverless compute, which starts in seconds. SQL and dashboards run on SQL warehouses with the Photon engine. And the table layout keeps queries reading less: liquid clustering keeps related rows in the same files, and predictive optimization compacts and vacuums managed tables for you.
Lesson complete
You finished “Batch processing with Spark”
Nicely done.