Machine Learning System Design Interview: Framework and 5 Worked Examples

“Design the recommendations for a video app’s home feed.” Two minutes later, the candidate is explaining transformer attention. The interviewer hasn’t heard what the feed is optimizing for, where the labels come from, or how the thing would serve results in under 200 milliseconds. That’s the most common way a machine learning system design interview goes sideways, and it has nothing to do with how much ML you know.

This round rewards structure. The interviewer wants to watch you turn a vague product goal into an ML problem, pick data and metrics that make sense, start simple, and then reason about serving, evaluation and what breaks later.

Below is the framework I’d use, then five worked examples you can practice out loud. The examples are illustrative, not answer keys, and any numbers in them are rough assumptions you’d state in the room. This post goes deep on the ML design round. For the whole MLE loop, see our guide to machine learning engineer interviews. If your role is about LLM apps, the LLM system design interview guide covers retrieval and guardrails.

How it differs from a regular system design round

A classic system design interview is about moving and storing data reliably: APIs, databases, caches, queues. Our system design interview template walks through that 45-minute flow.

The ML version keeps the infrastructure questions but adds a second layer. What exactly are you predicting? What counts as a positive label? How will you know the model is better before and after launch? What happens when user behavior shifts? You still need boxes and arrows. You just spend more of the hour on data, labels and metrics, and less on database sharding.

A framework for ML system design interviews

Seven steps. You won’t give each one equal time, and a good interviewer will pull you toward one or two of them.

1. Clarify the goal. What’s the product, who are the users, what’s the business metric? Then restate it as an ML task: “So we’re ranking videos for each user by predicted chance of a long watch.”
2. Pick metrics. One online metric the business cares about (watch time, conversion, fraud losses) and offline metrics you can compute on held-out data (AUC, precision at k, NDCG, MAE). Add a guardrail or two.
3. Data and labels. Where do training examples come from? Are labels explicit (a fraud chargeback) or implicit (a click)? How delayed and how noisy are they?
4. Features. User, item, context and interaction features. Say which ones need to be fresh and which can be computed in a daily batch.
5. Model. A simple baseline first, like a popularity list, logistic regression or gradient-boosted trees. Then what you’d upgrade to and why.
6. Serving. Batch or real-time predictions? Latency budget? For large catalogs, a candidate generation stage followed by a ranking stage.
7. Evaluate, launch, monitor. Offline evaluation, then an A/B test, then monitoring for drift, data pipeline breaks and feedback loops, plus a retraining plan.

Rough time split for a 45-minute round. The loop at the end is what interviewers mean by “what happens after launch?”

Here’s the thing most candidates miss: steps 1 to 3 are where you show judgment. If you rush them, every later choice looks arbitrary.

Example 1: Design a recommendation system for a video home feed

Clarify. Logged-in users, a catalog of a few million videos, a feed refreshed on each app open. Goal: long-term engagement, not raw clicks, because optimizing clicks alone tends to reward clickbait.

Metrics. Online: watch time per user and return visits over a week, with guardrails on hides and reports. Offline: recall at k for candidate generation, and AUC or NDCG for the ranker.

Data and labels. Impressions joined to what happened next. A positive might be “watched more than half” or “watched over 30 seconds.” Pick one and say why. Skips and hides are useful negatives.

Design. You can’t score millions of videos per request, so split it:

• Candidate generation pulls a few hundred to a few thousand videos from several sources: a two-tower embedding model with approximate nearest neighbor search, “users who watched this also watched,” fresh uploads from followed creators, and trending.
• Ranking scores those candidates with a heavier model, often predicting several things at once (click, watch time, like) and combining them into one score.
• Re-ranking applies business rules: diversity so the feed isn’t ten videos from one creator, freshness, and filtering anything removed or age-restricted.

Cold start. New users get popular and regional content plus a few onboarding choices. New videos get a small exploration budget so they collect enough signal to compete.

Follow-up to expect: “How do you avoid a feedback loop where the model only learns from what it already showed?” Log the position and the source of each impression, keep some exploration traffic, and be careful about treating “not shown” as “not liked.”

Example 2: Design search ranking for an online store

Clarify. Query in, ranked product list out. Business goal: purchases from search, without burying relevant items under sponsored ones.

Metrics. Offline: NDCG on judged query-product pairs or on click and purchase logs. Online: search conversion rate, plus the share of searches with no click, which is a strong frustration signal.

Design. Two stages again. Retrieval combines keyword search (BM25 on titles and descriptions) with embedding search so “running shoes” also finds “trainers.” Then a learning-to-rank model (gradient-boosted trees with a pairwise or listwise objective is a solid first pick) scores the top few hundred results using query-product text match, price, ratings, past click-through and stock.

The trap. Clicks are position-biased. Items at the top get clicked because they’re at the top. Mention it, and say how you’d correct for it: randomize a small slice of results, or model position as a feature during training and fix it at serving time.

Better NDCG offline doesn’t guarantee more purchases online. Plan for both checks.

Example 3: Design fraud detection for card payments

Clarify. Score each transaction in real time and decide: approve, decline, or step-up verification. Ask about the latency budget. Card authorization paths are tight, so assume tens of milliseconds for the model.

Labels. Chargebacks and confirmed fraud reports, which can arrive weeks later. That label delay shapes everything. Your training data always lags, and your newest fraud patterns are the least labeled.

Metrics. Accuracy is useless here, because fraud is a tiny share of transactions. Use precision and recall at the operating threshold, or fraud dollars caught per declined good transaction. Say that the threshold is a business decision: false declines lose customers, missed fraud loses money.

Design. Real-time features (number of transactions on this card in the last 10 minutes, distance from the last location, a new merchant for this card) come from a streaming feature store. Historical features come from batch. A gradient-boosted model is a strong baseline. Add hard rules for known patterns, and send borderline cases to verification or manual review.

Follow-up to expect: “Fraudsters adapt. How do you keep up?” Monitor score distributions and decline rates daily, retrain often, and use fast proxy labels (a customer disputing within 48 hours) alongside the slow ground truth.

Example 4: Design ETA prediction for food delivery

Clarify. Predict when an order will arrive, shown when the user orders and updated along the way. Is it one number, or a range?

Metrics. MAE in minutes is the obvious one. But users hate late more than early, so also track the share of orders that arrive later than promised. That pushes you toward predicting a range or a high quantile instead of just the average.

Design. Break the ETA into parts: restaurant prep time, courier pickup, and travel. Each has different features. Prep time depends on the restaurant, the order size and how busy the kitchen is right now. Travel depends on distance, traffic and time of day. A model per part (or one model with all features) trained on completed orders, retrained frequently, with real-time features for kitchen load and courier location.

You’ll probably get the rainy-Friday question: “What happens on a rainy Friday night?” Demand spikes, couriers slow down, and a model trained on normal days runs late. Weather and live order volume as features, plus monitoring of error by hour, is the honest answer.

Example 5: Design harmful content detection for a social app

Clarify. Which harms (spam, nudity, harassment, violent content)? Text, images, video? What happens on a positive: removal, reduced distribution, or human review?

Metrics. Precision and recall per harm type, measured against human-labeled samples. For the business: prevalence of harmful content that users actually see. Track false removals and appeals as a guardrail.

Design. Cheap filters run on every upload (hash matching against known bad content, simple classifiers). Heavier models run on what’s left. High-confidence violations get removed automatically. Uncertain ones go to a human review queue, prioritized by predicted severity and reach. Reviewer decisions flow back as training labels.

The trap. Different thresholds per harm type. Auto-removing spam at moderate confidence is fine. Auto-removing something that might be news coverage of violence needs a much higher bar or a human.

Follow-up questions that come up in almost every ML design round

Whatever the prompt, expect a few of these, and have a short answer ready for each:

• How would you handle a model that does well offline and flat in the A/B test?
• What’s your baseline, and why start there?
• How do you keep training and serving features consistent?
• How would you detect data drift, and what would you do about it?
• How often would you retrain, and what triggers it?
• How would you explain a prediction to a non-technical stakeholder?

The training-serving skew question trips people up. A clean answer: compute features with the same code path for training and serving, usually through a feature store, and log the features actually used at serving time so you can train on exactly what the model saw.

“What happens six months after launch?” is a question you can rehearse before anyone asks it.

Practicing with OfferBull

ML system design is a talking round. You can know every concept above and still lose the thread when someone interrupts with “wait, where do the labels come from?” The only fix I know is rehearsing out loud with interruptions.

OfferBull is an AI mock interview app for iOS and Android. An interviewer voice asks you questions out loud and you answer by speaking. Paste the job description or upload your resume, and the questions follow the role, so a recommendations team gets different prompts than a trust and safety team. It asks follow-up questions based on your answers and your resume. Say “two-tower model” and it may ask how you’d pick negatives. During practice, you can see answer suggestions, and STAR structures for the behavioral questions that usually share the loop. You can practice in English or Chinese.

It’s built for the weeks before the interview. On the day, it’s your framework and your voice.

Start an AI mock interview

FAQ

How long is an ML system design interview? Usually 45 to 60 minutes. Plan on roughly 10 to 15 minutes for goal, metrics and data, 20 to 25 for features, model and serving, and the rest for evaluation, monitoring and follow-ups.

Do I need to write code in this round? Usually not. You might sketch a loss function or pseudo-code for a feature, but it’s mostly whiteboard and conversation.

Should I propose deep learning right away? No. Start with a baseline you can defend, then explain what a more complex model would buy you and what it would cost in latency, data and maintenance.

How is this different from an LLM system design interview? LLM rounds focus on prompting, retrieval, evaluation of generated text and safety layers. Classic ML design is more about labels, features, ranking and offline-to-online metrics. Some loops mix both.

---

Practice This Out Loud

Pick Example 1 and talk through it in 35 minutes with a timer, using the seven steps. Then do Example 3 in 20, spending at least five minutes on labels and thresholds. If you can handle “where do the labels come from?” without pausing, try it in a voice mock interview that asks follow-ups. → Practice Now