Sample questions
Aimlops Model Monitoring Drift DetectionDifficulty 1
In production model monitoring, what is 'data drift'?
- aA drop in the model's accuracy caused only by a bug in the serving code, unrelated to input data.
- bA change in the statistical distribution of the model's input features compared to training data.✓
- cThe gradual increase in inference latency as request volume grows over time.
- dA change in the relationship between input features and the target label, with inputs unchanged.
Explanation:Data drift refers to the input feature distribution shifting away from what the model was trained on (e.g., average order value rising over time), independent of whether the input-output relationship itself changed. (a) describes a code defect, not a distributional property. (c) describes a latency/performance concern, not statistics of the data. (d) describes concept drift, the input-output relationship changing, which is a distinct phenomenon.
Aimlops Model Monitoring Drift DetectionDifficulty 1
What is 'concept drift', and how does it differ from data drift?
- aConcept drift means the feature distribution changed; data drift means the label distribution changed, which does not hold in practice.
- bConcept drift and data drift are two names for the exact same phenomenon, used interchangeably.
- cConcept drift only happens in computer vision models, while data drift only happens in tabular models.
- dConcept drift means the true relationship between inputs and the target changed, even if input distributions look stable.✓
Explanation:Concept drift is a change in P(label | input) — the underlying pattern the model is supposed to capture shifts (e.g., what counts as 'fraud' changes), which can happen even when the inputs themselves look statistically unchanged. (a) swaps the definitions. (b) treats two distinct phenomena as one, which loses an important diagnostic distinction. (c) invents a modality restriction that doesn't reflect how either concept is actually scoped.
Aimlops Model Monitoring Drift DetectionDifficulty 2
A fraud-detection model's input features (transaction amount, merchant category, time of day) look statistically the same as during training, but the model's precision drops sharply because fraudsters changed their tactics. What is this most likely an example of?
- aConcept drift: the relationship between the same-looking inputs and the fraud label has changed.✓
- bData drift: the input feature distributions have shifted away from the training distribution.
- cA labeling bug: the ground-truth fraud labels are being recorded with the wrong data type.
- dModel overfitting: the model memorized the training set and never generalized to new transactions.
Explanation:Since inputs look unchanged but the pattern connecting them to the label ('is this fraud?') has shifted, this is concept drift (a). (b) is ruled out by the premise that feature distributions look the same. (c) and (d) invent unrelated failure modes (a data-type bug, or a training-time generalization problem) that the scenario gives no evidence for.
Aimlops Model Monitoring Drift DetectionDifficulty 1
For a numeric feature like 'customer age', which of these is a typical thing to monitor to detect data drift, without needing ground-truth labels?
- aThe number of unit tests that currently pass in the model's training repository.
- bThe exact git commit hash of the code that trained the currently deployed model.
- cHow many milliseconds the model takes to return a prediction for a single request, though this does not reflect how the mechanism actually behaves in practice.
- dHow the distribution (e.g., mean, quantiles, histogram) of incoming values compares to the training distribution.✓
Explanation:Feature-distribution monitoring compares live input statistics (mean, quantiles, histogram shape) against the training baseline, which needs no ground-truth labels (d). (a) and (b) are software/repo metadata unrelated to input statistics. (c) measures latency, a serving-performance concern, not a distributional one.
Aimlops Model Monitoring Drift DetectionDifficulty 2
Why is monitoring for data drift useful even before a model's downstream performance metric (like accuracy) has visibly degraded?
- aIt can act as an early warning signal, since input shifts often precede a measurable drop in performance.✓
- bIt replaces the need to ever measure real performance metrics, since drift scores are always more accurate.
- cIt is required by every cloud provider's terms of service before a model can be deployed to production.
- dIt guarantees the model's predictions remain correct as long as no drift is detected in any feature.
Explanation:Distribution shifts in inputs often show up before enough labeled outcomes accumulate to move a lagging performance metric, so drift monitoring gives an earlier, complementary signal (a). (b) wrongly claims drift monitoring replaces performance evaluation rather than supplementing it. (c) invents a vendor requirement. (d) overclaims a guarantee that drift monitoring cannot provide, since accuracy also depends on factors beyond input distribution.
Aimlops Model Monitoring Drift DetectionDifficulty 2
A monitoring dashboard shows 'model performance decay' over the last month. What does this term describe?
- aThe gradual reduction in the physical storage size of the model artifact file on disk.
- bA gradual decline in the model's real-world predictive quality (e.g., accuracy, precision) compared to when it launched.✓
- cThe scheduled removal of old model versions from the model registry to save storage cost, though this does not reflect how the mechanism actually behaves in practice.
- dAn increase in the number of feature columns the model receives at inference time.
Explanation:Performance decay describes predictive quality degrading over time relative to launch, usually driven by drift or a changing environment (b). (a) describes disk usage, unrelated to predictive quality. (c) describes registry housekeeping, not a quality signal. (d) describes a schema change to the feature set, which is a possible cause of drift but not the definition of decay itself.