Sample questions
Ml Problem FramingDifficulty 1
Why is a dataset typically split into separate train, validation, and test sets?
- aTo make the training loop run faster by processing smaller chunks of data one at a time
- bTo fit the model on one part, tune choices on another, and get an unbiased read on unseen data✓
- cTo reduce the total amount of data that needs to be stored on disk during the project
- dTo let the model train on every example twice so it memorizes the patterns more reliably
Explanation:The train set fits model parameters, the validation set is used to make decisions (which model, which settings) without touching the test set, and the test set gives a final, unbiased read on generalization because it never influenced any decision. (a) and (c) describe storage/speed side-effects, not the actual reason. (d) is wrong: repeating exposure to the same data risks memorization (overfitting), which is exactly what the split is meant to detect.
Ml Problem FramingDifficulty 1
In ML problem framing, what is a "baseline model"?
- aThe most complex model architecture the team is technically capable of building for the problem
- bA model that has already been deployed to production and is being replaced by a new candidate
- cA simple, cheap-to-build rule used as a reference point for judging a more complex model✓
- dThe final model version chosen after all experimentation and hyperparameter tuning has finished
Explanation:A baseline is intentionally simple (a rule of thumb, a majority-class guess, a basic linear model) so it is fast to build and gives a reference score. Every later, more complex model must beat it to justify its added cost and complexity. (a) is the opposite of the idea (baselines are simple, not the fanciest option). (b) confuses baseline with an existing production model, which is a different concept (a champion to beat, not necessarily simple). (d) describes the final chosen model, not the starting reference point.
Ml Problem FramingDifficulty 2
A team skips building a baseline and goes straight to a complex model that reaches 82% on their chosen metric. Why is this risky?
- aWithout a reference score, 82% cannot be judged as good, mediocre, or barely above a trivial rule✓
- bComplex models always score lower than baselines, so 82% is almost certainly an error in the pipeline
- cSkipping a baseline is only risky for regression problems, not for classification problems like this one
- dIt is risky because complex models take longer to train, which has nothing to do with the metric value
Explanation:A metric value is only interpretable relative to something: a trivial rule (predict the majority class), a simple heuristic, or a prior model. If a majority-class guess already scores 80% on this data, the complex model added almost nothing despite its cost. (b) is a false claim — nothing guarantees complex models score lower. (c) is wrong: the risk applies to any ML problem type. (d) mixes up training cost with the separate issue of not knowing if the score itself is good.
Ml Problem FramingDifficulty 1
What does the "accuracy" metric measure in a classification problem?
- aThe fraction of predicted positives that are actually positive
- bThe fraction of actual positives that the model successfully identifies
- cThe model's ability to rank positive examples above negative examples
- dThe fraction of all predictions (positive and negative) that match the true label✓
Explanation:Accuracy = (correct predictions) / (total predictions), counting both classes. (a) describes precision. (b) describes recall. (c) describes the ranking notion behind AUC-ROC, not accuracy. Accuracy treats every prediction the same regardless of class, which is exactly why it becomes misleading when classes are imbalanced.
Ml Problem FramingDifficulty 2
What does "precision" answer, in plain terms, for a classifier?
- aOf all the examples that are truly positive, how many did the model catch?
- bOf all the examples the model flagged as positive, how many are actually positive?✓
- cAcross all possible decision thresholds, how well does the model separate the two classes?
- dHow many predictions in total, positive and negative combined, were correct?
Explanation:Precision = true positives / (true positives + false positives): among what the model called positive, how much of it was right. (a) describes recall (coverage of actual positives). (c) describes a threshold-independent ranking measure like AUC-ROC. (d) describes accuracy. Precision is specifically about trusting a positive alarm once it fires.
Ml Problem FramingDifficulty 2
What does "recall" answer, in plain terms, for a classifier?
- aOf everything the model flagged as positive, how much was actually positive?
- bHow balanced are the predicted classes compared to the true class distribution?
- cOf all the examples that are truly positive, how many did the model actually find?✓
- dHow consistent are the model's predictions across repeated runs on the same input?
Explanation:Recall = true positives / (true positives + false negatives): of everything that was truly positive, how much did the model manage to catch. (a) describes precision. (b) describes a class-balance property unrelated to recall's definition. (d) describes run-to-run stability, a different concern entirely (reproducibility, not recall).