yoklainterview sim
ProgressHistory

Data Engineer Pe Spark Api Semantics Udfs Interview Questions

75 verified Data Engineer Pe Spark Api Semantics Udfs interview questions — solve with answers, learn from explanations, test yourself in a real simulation.

Try the real simulation →

Sample questions

Pe Spark Api Semantics UdfsDifficulty 2
In Spark's Catalyst optimizer, a DataFrame query goes through several plan stages before execution. What is the correct order?
  • aPhysical plan → optimized logical plan → logical plan
  • bLogical plan → optimized logical plan → physical plan✓
  • cOptimized logical plan → logical plan → physical plan
  • dPhysical plan → logical plan → optimized logical plan
Explanation:Catalyst first builds a logical plan from the DataFrame/SQL code, applies rule-based optimizations to produce the optimized logical plan, and finally generates the physical plan that the execution engine runs.
Pe Spark Api Semantics UdfsDifficulty 1
df.filter(df.amount > 100).explain()

Calling .explain() with no arguments on a DataFrame prints which plan by default?
  • aOnly the physical plan✓
  • bOnly the unresolved logical plan
  • cAll four plan stages (parsed, analyzed, optimized, physical)
  • dOnly the optimized logical plan
Explanation:explain() with no arguments (extended=False) prints just the physical plan. Passing extended=True or a mode string like 'formatted' shows more detail, including the logical plan stages.
Pe Spark Api Semantics UdfsDifficulty 2
In Catalyst's optimized logical plan, what does 'column pruning' do?
  • aRemoves rows that fail a filter condition before scan
  • bMerges two scans of the same table into one
  • cReorders join operations by estimated cost
  • dDrops columns never referenced downstream✓
Explanation:Column pruning finds which columns are actually used later in the plan and rewrites the scan to read only those columns, reducing I/O especially for columnar formats.
Pe Spark Api Semantics UdfsDifficulty 2
What does Catalyst's constant folding optimization do with an expression like WHERE price > 10 + 5?
  • aIt pushes the whole expression down to the storage layer unchanged
  • bIt rewrites the comparison into a join condition
  • cComputes 10 + 5 at plan time and substitutes 15✓
  • dIt converts the filter into a window function
Explanation:Constant folding evaluates expressions made only of literals during query planning, so the physical plan compares price > 15 directly instead of recomputing 10 + 5 for every row.
Pe Spark Api Semantics UdfsDifficulty 2
What does 'predicate pushdown' mean when Spark reads from a filterable data source?
  • aFilters are moved to run after the join stage instead of before
  • bThe filter is passed to the source to skip non-matching data early✓
  • cFilters are converted into UDFs for flexibility
  • dAll filters are evaluated only on the driver
Explanation:With predicate pushdown, Spark translates a filter condition into something the source (e.g. JDBC, Parquet) can apply itself, so fewer rows or files are actually read into the cluster.
Pe Spark Api Semantics UdfsDifficulty 2
A team writes:
df.filter(is_valid_udf(df.email))

where is_valid_udf is a Python UDF, then reads explain(). What should they expect regarding pushdown to the underlying Parquet source?
  • aThe UDF filter is automatically pushed down because Parquet supports arbitrary filters
  • bCatalyst rewrites the UDF into a built-in expression before pushdown
  • cPushdown still happens because Catalyst inlines the UDF bytecode
  • dThe UDF filter cannot be pushed down; Catalyst evaluates it after the scan✓
Explanation:Catalyst can only push down filters it understands as expressions. A UDF is opaque to the optimizer, so the filter it's part of is evaluated inside Spark after rows are read, not pushed into the Parquet reader.

By specialization

Pipelines / ETL 450Streaming 450Warehousing 450

By seniority

Data Engineer JuniorData Engineer MidData Engineer Senior

Other fields

BackendFrontendFullstackDevOps / CloudAI EngineerDatabaseMobileSecurityQA / Test AutomationGame DevelopmentML Engineer

Test yourself against the 1950-question Data Engineer bank.

Start interview