Sample questions
Pe Spark Execution Shuffle PartitionsDifficulty 1
In Spark, when you call df.filter(...).select(...), what happens immediately?
- aThe filter and select run right away and return an in-memory result.
- bSpark launches a job on the cluster to validate the schema.
- cSpark builds a logical plan for the transformations; nothing runs yet.✓
- dSpark writes an execution log to disk before continuing.
Explanation:Transformations like filter and select are lazy in Spark; they only build up a logical plan. Execution starts when an action such as count(), collect(), or write() is called.
Pe Spark Execution Shuffle PartitionsDifficulty 1
Which of the following is an action rather than a transformation in Spark?
- a
df.count()✓ - b
df.withColumn(...) - c
df.groupBy(...) - d
df.repartition(10)
Explanation:count() triggers execution of the accumulated plan and returns a result to the driver, which is the defining property of an action. withColumn, groupBy, and repartition only add steps to the logical plan.
Pe Spark Execution Shuffle PartitionsDifficulty 2
In Spark's execution hierarchy, what triggers the creation of a new job?
- aCalling a transformation such as
map - bReaching a shuffle boundary
- cThe driver starting up
- dCalling an action such as
collect()✓
Explanation:A job is created each time an action is called; the job is then split into stages, and each stage into tasks (one per partition).
Pe Spark Execution Shuffle PartitionsDifficulty 2
What determines where one stage ends and the next stage begins in a Spark job?
- aThe number of executors available
- bA shuffle boundary✓
- cThe size of the driver's memory
- dThe order the code was written in
Explanation:Spark splits a job into stages at shuffle boundaries. A wide dependency (e.g. groupBy, join) requires data to move across partitions, so it forces a new stage.
Pe Spark Execution Shuffle PartitionsDifficulty 2
Which operation is a narrow dependency, meaning each output partition depends on a known, limited set of input partitions?
- a
map()✓ - b
groupByKey() - c
join() without a broadcast hint - d
repartition()
Explanation:map() transforms each input partition independently into one output partition — a narrow dependency. groupByKey, join, and repartition require shuffling data between partitions (wide dependency).
Pe Spark Execution Shuffle PartitionsDifficulty 1
What is a shuffle in Spark?
- aRandomly reordering rows within a single partition
- bCompressing a DataFrame before writing it to disk
- cRestarting a failed executor
- dRedistributing data across partitions over the network✓
Explanation:A shuffle redistributes data so that rows with related keys end up in the same output partition, which typically means moving data across the network between executors.