Sample questions
Ml Monitoring DriftDifficulty 1
In the context of a deployed machine learning model, what does the term 'data drift' refer to?
- aA bug in the training pipeline that causes the model to be trained on corrupted labels.
- bThe gradual decrease in a model's inference latency as the serving infrastructure ages.
- cA shift in the input feature distribution compared to the data the model was trained on.✓
- dA deliberate change to the model's architecture made by the engineering team after deployment.
Explanation:Data drift specifically means the input distribution (feature values) seen in production has shifted away from the training distribution, even if the true relationship between inputs and outputs hasn't changed. It's distinct from pipeline bugs, latency issues, or architecture changes.
Ml Monitoring DriftDifficulty 1
What does 'concept drift' mean for a deployed model?
- aThe relationship between inputs and the correct output changes over time.✓
- bThe set of features available to the model is reduced due to a schema change in the source database.
- cThe model's output format changes from a probability score to a class label.
- dThe training dataset is replaced with a larger dataset covering more regions.
Explanation:Concept drift is about the function being learned (the relationship between inputs and outcome) changing, not just the input distribution. This is why a model can keep seeing 'normal-looking' inputs yet make increasingly wrong predictions.
Ml Monitoring DriftDifficulty 2
A team notices their model's accuracy has dropped even though the distribution of input feature values looks nearly identical to the training period. Which type of drift is the most likely explanation?
- aData drift — the input feature distribution has shifted in a way that is not detectable by summary statistics.
- bLabel leakage — the training set accidentally included the target variable as a feature.
- cSampling bias — the production traffic represents a different population than the test set used for evaluation.
- dConcept drift — the relationship between inputs and the correct outcome has changed, not the inputs themselves.✓
Explanation:If input distributions look stable but predictive performance drops, the mapping from input to output has likely changed (concept drift) rather than the inputs shifting (data drift).
Ml Monitoring DriftDifficulty 1
What best describes 'offline' model performance monitoring?
- aContinuously logging every prediction request and response in real time without ever computing metrics.
- bPeriodically evaluating the model against labeled recent data to compute accuracy-style metrics.✓
- cMonitoring only the CPU and memory usage of the serving infrastructure.
- dWatching the model's training loss curve during the original training run.
Explanation:Offline monitoring means recomputing supervised metrics (precision, recall, error rate, etc.) once ground-truth labels for recent production data become available, typically on a schedule such as daily or weekly.
Ml Monitoring DriftDifficulty 2
Since ground-truth labels are often not immediately available in production, what does 'online' monitoring typically rely on instead?
- aProxy signals at prediction time, such as input and prediction distributions.✓
- bA manual review of every single prediction by a human expert before it is served.
- cRe-running the full offline evaluation suite on every incoming request.
- dWaiting for the next scheduled retraining cycle before checking model health.
Explanation:Online monitoring must work without immediate ground truth, so it tracks proxies like input/prediction distribution shifts, confidence scores, and volume anomalies to catch problems before labeled evaluation is possible.
Ml Monitoring DriftDifficulty 2
A fraud-detection model's true labels (confirmed fraud or not) only become available 30 days after a transaction, once a dispute process resolves. What is the main monitoring challenge this creates?
- aThe model must be retrained every 30 days regardless of whether performance has changed.
- bThe feature store cannot store data older than 30 days, so historical comparisons are impossible.
- cPrecision and recall can only be computed with a 30-day lag, slowing problem detection.✓
- dThe transactions must be reprocessed through the model a second time after the labels arrive.
Explanation:When ground truth is delayed by a fixed window (here 30 days), teams need proxy or online monitoring to detect issues in near real time, since label-based offline metrics inherently lag by that same window.