PySpark and Apache Spark Interview Questions for Data Engineers

The interviewer pastes six lines of PySpark into the shared doc and asks two things: how many jobs does this kick off, and where’s the shuffle? You’ve written code like it a hundred times. You’ve never had to narrate what Spark does with it, line by line, while someone watches.

That’s what most PySpark interview questions really test. Can you explain what happens under the API? Syntax matters less than people think. Interviewers assume you can look up withColumn. They want to know if you’ll notice the join that’s about to move 2 TB across the network.

Below are the Spark questions data engineers get asked most, from core concepts to skew, streaming, pipeline design and behavioral stories. Sample answers are illustrative, so check your reasoning against them and then say it your own way. If you want the wider picture of the data engineering loop (SQL, modeling, the coding round), start with our guide on how to prepare for data engineering interviews and come back here for the Spark part.

What a Spark interview usually looks like

The Spark portion of a data engineering loop usually mixes concept checks, a PySpark coding problem (often a dedupe or ranking), a tuning scenario where a job got slow or ran out of memory, and a pipeline design question. Senior roles lean harder on the last two. Either way, the follow-up is where people stumble. “Okay, and why is that a shuffle?” is a very common second question.

Core Spark concepts

1. What’s the difference between a transformation and an action?

Sample answer: “Transformations like filter, select and groupBy describe a new dataset but don’t compute anything. Spark is lazy, so it just records the plan. Actions like count, collect or write need a result, so they’re what actually trigger execution. The upside of laziness is that Spark sees the whole plan before running it, so the optimizer can push filters down and prune columns.”

2. Walk me through this code. How many jobs, and where’s the shuffle?

from pyspark.sql import functions as F

df = spark.read.parquet("s3://bucket/events/")
us = df.filter(F.col("country") == "US").select("user_id", "amount")
totals = us.groupBy("user_id").agg(F.sum("amount").alias("total"))
totals.cache()
print(totals.count())
totals.write.mode("overwrite").parquet("s3://bucket/user_totals/")

Here’s how I’d say it: “There are two actions, count and write, so at least two jobs. The UI may show a couple more for things like file listing, so I wouldn’t promise an exact number. The filter and select are narrow, they run partition by partition. The groupBy needs all rows for a user in one place, so that’s a shuffle and a stage boundary. cache() is lazy too. The first action, count, fills the cache, and the write then reads totals from the cache instead of rereading S3.”

Hedging on the exact job count is fine. Precision about what you know beats a confident wrong number.

3. Job, stage, task. What’s the difference?

An action creates a job. Spark splits the job into stages at shuffle boundaries. Each stage runs as a set of tasks, one task per partition. So if a stage reads 400 partitions, you get 400 tasks, spread across the executors’ cores.

4. Narrow vs. wide transformations?

In a narrow transformation (filter, select, withColumn, map), each output partition depends on one input partition, so no data moves. In a wide one (groupBy, join, distinct, repartition), output partitions need data from many input partitions, so Spark has to shuffle. Shuffles mean disk I/O, serialization and network traffic, which is why the Apache Spark docs call them a “complex and costly operation.”

One action, one job. Narrow steps share a stage; the shuffle starts a new one.

5. RDD vs. DataFrame vs. Dataset?

“An RDD is the low-level collection: full control, no optimizer. A DataFrame has a schema, so the Catalyst optimizer can rewrite the query. A Dataset is the typed version, but that’s Scala and Java only. In PySpark I work with DataFrames and only drop to RDDs when the DataFrame API can’t express something.”

6. What happens when you call collect() on a big DataFrame? Every row comes back to the driver, and if it doesn’t fit in driver memory, the driver dies. Use take(n) to look, and write results out instead of pulling them back.

PySpark coding questions

Expect at least one hands-on problem. The two below test window functions, where many candidates get shaky.

7. Keep only the latest record per user.

from pyspark.sql import Window, functions as F

w = Window.partitionBy("user_id").orderBy(F.col("updated_at").desc())
latest = (df.withColumn("rn", F.row_number().over(w))
            .filter(F.col("rn") == 1)
            .drop("rn"))

Mention ties. If two rows share an updated_at, row_number picks one arbitrarily, so add a second sort column if the result must be deterministic.

8. Top three products by revenue in each category. Same pattern, but ask how to handle ties first. row_number gives exactly three rows per category; rank or dense_rank can give more. Asking is part of the answer.

9. Why is this Python UDF slow, and what would you use instead?

A regular Python UDF runs row by row in a Python worker, so Spark has to ship data from the JVM to Python and back, and the optimizer can’t see inside it. First choice: rewrite it with built-in functions from pyspark.sql.functions. If you really need Python logic, a pandas UDF processes batches through Apache Arrow and is usually much faster than a row-at-a-time UDF.

10. After a join, my row count went up. Why?

The join key isn’t unique on one side. If a user appears twice in the dimension table, each of their events is duplicated. Check with groupBy(key).count().filter("count > 1") before joining. It’s the bug that quietly double-counts revenue.

Performance and tuning questions

This is where senior candidates stand out.

11. What’s data skew, and how do you fix it?

Skew is when a few keys hold most of the data, so one partition is huge and one task runs for an hour while the other 199 finished in a minute. You spot it in the Spark UI: one task’s duration and shuffle read are way above the median.

Fixes, roughly in the order I’d try them:

• Check that Adaptive Query Execution is on. It’s enabled by default since Spark 3.2.0, and its skew join handling splits oversized partitions in sort-merge joins.
• If the other side of the join is small, broadcast it so the big side never shuffles.
• Filter or handle the hot key separately. Null keys are a classic culprit.
• Salt the key: add a random suffix on the skewed side, replicate the other side across the suffixes, join, then aggregate back.

Draw the skew before you fix it. One giant partition makes the problem obvious to everyone at the table.

12. When does Spark use a broadcast join?

When one side’s estimated size is under spark.sql.autoBroadcastJoinThreshold, which defaults to 10 MB. You can force it with F.broadcast(small_df) in the join or a BROADCAST hint in SQL. The catch: the broadcast table has to fit in memory on the driver and every executor, so “just broadcast it” stops working as the dimension table grows.

13. repartition vs. coalesce?

repartition(n) does a full shuffle and gives evenly sized partitions, up or down. coalesce(n) only reduces partitions by merging existing ones, so it avoids a full shuffle. I’d use coalesce to cut output files at the end of a job and repartition when the data really needs redistributing.

14. What’s spark.sql.shuffle.partitions, and should you change it?

It’s the number of partitions after a shuffle for joins and aggregations, 200 by default. On a big job that can mean huge partitions that spill to disk; on a tiny one, 200 near-empty tasks. With AQE on, Spark coalesces small post-shuffle partitions at runtime, so you can start generous and let it trim.

15. When would you cache a DataFrame?

Only when you reuse it in more than one action and recomputing it is expensive. For DataFrames, cache() uses MEMORY_AND_DISK_DESER by default (the RDD default is MEMORY_ONLY). Call unpersist() when you’re done. Caching everything “to be safe” just pushes useful data out of memory.

A debugging scenario you should rehearse

“A daily job usually takes 20 minutes. Today it took three hours and nobody changed the code. Walk me through it.”

They want an order of operations. Something like:

“First I’d check whether the input changed. Bigger partition for today, a new upstream source, a schema change. Then I’d open the Spark UI and find the slow stage. If one or two tasks are taking far longer than the median, that’s skew, so I’d look at which key they’re processing. If all tasks are slow and I see a lot of spill, partitions are too big, so I’d look at shuffle partitions and memory. I’d also run explain() and compare the plan with a good day. A broadcast join that turned into a sort-merge join because the dimension table crossed the threshold would explain a sudden jump like this. Once I know the cause, I’d fix that one thing, not five knobs at once.”

Structured Streaming questions

If the job description mentions Kafka, real-time or streaming, expect a few of these.

16. What’s a watermark? It tells Spark how late event-time data can arrive and still be counted, for example withWatermark("event_time", "10 minutes"). It lets Spark drop old aggregation state instead of keeping it forever. Data arriving later than the watermark allows may be dropped.

17. What are the output modes? Append (only new rows), update (rows that changed since the last trigger) and complete (the whole result table every time). Not every mode works with every query.

18. How does Spark get exactly-once results? Checkpointing plus a write-ahead log track what each micro-batch processed. That gives end-to-end exactly-once only if the source can replay (Kafka offsets, files) and the sink is idempotent or transactional. Say both conditions. Candidates often stop at “checkpointing.”

A pipeline design question

“Design a daily batch job that turns about 2 TB of raw clickstream events into a user-level summary table for analysts.”

I’d structure the answer around five decisions. Input layout first: Parquet partitioned by event date, so the job reads one day, not the whole history. Then late data: either reprocess a rolling window of recent dates or accept a cut-off and say so. Third, idempotency. Write with dynamic partition overwrite (spark.sql.sources.partitionOverwriteMode set to dynamic) so rerunning a day replaces that day instead of appending duplicates. Fourth, data quality: row counts against the source, null checks on keys, a uniqueness check on user_id in the output. Last, file size. Coalesce before writing so analysts don’t query 40,000 tiny files.

If the interviewer pushes toward the cloud side (S3 layouts, EMR vs. Glue, IAM for the job), our AWS interview questions guide covers that half. For the SQL your analysts will run on top, see SQL and database interview prep.

Behavioral questions for data engineers

Data engineers tend to under-prepare these. Common ones:

• Tell me about a pipeline that broke in production. What did you do?
• Describe a time you made a slow job much faster.
• Tell me about a disagreement with an analyst or data scientist about data definitions.
• Describe a time bad data reached a dashboard before you caught it.

Two illustrative answers. The details are made up to show the shape.

“Tell me about a pipeline that broke in production.”

• Situation: I was on a three-person data platform team. Our nightly orders job, which fed the finance dashboard, failed two nights in a row with executor out-of-memory errors.
• Task: I was on call, and finance needed the numbers before their 9 a.m. review.
• Action: I reran yesterday’s partition alone to get the dashboard filled, then dug into the Spark UI. One task in the join stage was reading roughly 40 times more data than the median. The key was a null customer_id from a new guest-checkout flow upstream. I split null keys out, processed them separately, and added a check that alerts when nulls in that column jump.
• Result: The job went back to its normal runtime and finance had their numbers on time. I also wrote a short note for the upstream team so guest orders got a proper placeholder ID.

“Describe a time you made a slow job much faster.”

“We had a feature table job that took about four hours and kept missing its SLA. Before touching configs, I read the plan. It was joining a big events table to a product table that had grown to a few hundred MB, so it was doing a sort-merge join with a big shuffle every night. Most of the product columns weren’t even used. I selected the four columns we needed, which shrank it enough to broadcast with a hint, and filtered events to the last 30 days before the join instead of after. It dropped to under an hour. The part I’d do again is reading the plan before turning knobs.”

Pick one broken-pipeline story and tell it out loud until the order of events comes out clean.

If you need help picking and shaping stories, the STAR method guide has more examples and timing tips.

Practicing with OfferBull

You can read about shuffles all week and still freeze at “why is that a stage boundary?” Saying it out loud, with follow-ups, is what fixes that.

OfferBull is an AI mock interview app for iOS and Android. An interviewer voice asks you questions out loud and you answer by speaking. Paste the job description or upload your resume, and the questions follow the stack in front of you, so a role built on Spark and Kafka gets different questions than one built on dbt and a warehouse. It asks follow-ups based on your answers and your resume. Mention salting and it may ask how you’d pick the number of salt buckets. When you’re stuck during practice, it can show answer suggestions and a STAR structure for behavioral questions. You can practice in English or Chinese.

It’s for the prep weeks. In the interview itself, it’s just you.

Start an AI mock interview

FAQ

Do I need to know Scala for Spark interviews? Usually not, if the job description says PySpark. You should still know that Datasets are a Scala and Java feature and that Python UDFs cost more than built-in functions, because those differences come up.

How deep do Spark internals questions go? For most data engineering roles: jobs, stages, tasks, shuffles, partitions, joins, caching and AQE. Memory tuning details come up more for senior or platform roles.

Should I practice on a real cluster? A local pyspark session is enough for coding questions. For tuning questions, run something big enough to shuffle and spend time in the Spark UI, because “what would you look at in the UI?” is a very common follow-up.

Practice This Out Loud

Tonight, take the six-line snippet above and explain it out loud in under two minutes: jobs, stages, the shuffle, the cache. Then answer “the job took three hours today, why?” in three. If both come out smoothly, try them in a voice mock interview that asks follow-ups. → Practice Now