Sample questions
Aimlops Deployment Strategies MlDifficulty 1
What does a 'canary deployment' mean specifically for an ML model release, compared to a generic canary deployment for regular application code?
- aIt only checks that the serving container starts and passes a basic health check, the same as any other service.
- bA small percentage of live traffic is routed to the new model version while model-specific signals (prediction distribution, business KPI, error rate) are monitored, not just infrastructure health.✓
- cTwo identical production environments run side by side and all traffic is switched to the new one instantly.
- dThe new model version is only ever evaluated offline against a static test set, and never receives live traffic, so it provides no information about how the model behaves under real production conditions or load.
Explanation:A model canary reuses the general canary mechanic (small live-traffic slice) but the signals monitored are model-specific: prediction/label distribution shifts and business KPIs, not just CPU/memory or HTTP status. (a) describes generic infra health checks only. (c) describes blue-green. (d) describes offline evaluation, which is not a canary at all since it never touches live traffic.
Aimlops Deployment Strategies MlDifficulty 1
What is 'shadow deployment' (shadow mode) for an ML model?
- aThe new model receives a small percentage of live traffic and its response is returned directly to those users.
- bTwo full production environments are swapped instantly, and all traffic moves to the new one at once.
- cInstances running the old model are replaced gradually, a few at a time, by instances running the new model.
- dThe new model runs in parallel on a copy of live production requests, its predictions are logged for comparison, but the response actually served to users always comes from the current production model.✓
Explanation:In shadow mode, the candidate model receives (a copy of) real traffic purely for offline comparison; it never affects what the user sees, so there is zero direct risk to users from its output. (a) describes canary, where the new model's output is actually served. (b) describes blue-green. (c) describes rolling update.
Aimlops Deployment Strategies MlDifficulty 2
For ML systems, blue-green deployment has an extra requirement beyond a generic blue-green cutover of application code. What is it?
- aThe model artifact and its paired preprocessing/feature-transformation code must be switched together atomically, so serving never pairs a new model with mismatched (old) feature logic.✓
- bOnly the model weights are ever switched; the preprocessing code is expected to stay unchanged forever across all model versions.
- cBlue-green for ML means the model can only be deployed on CPU-based infrastructure, never GPU.
- dBlue-green for ML requires maintaining two separate copies of the training dataset, unrelated to the serving environment, making it a purely data-management concern with no bearing on cutover.
Explanation:A model and the feature/preprocessing code it was trained against form one coupled unit; cutting over the model without cutting over matching preprocessing risks train-serve skew, so both must switch atomically. (b) wrongly assumes preprocessing never changes. (c) and (d) describe constraints unrelated to what blue-green actually requires for ML.
Aimlops Deployment Strategies MlDifficulty 1
At a basic level, what is a model A/B test?
- aRunning the same model twice on the same input to check whether it is deterministic.
- bComparing a model's accuracy on a training set versus a validation set before deployment.
- cSplitting users or traffic randomly into disjoint groups, each served by a different model version, and comparing an outcome metric between the groups.✓
- dDeploying a model to two different cloud regions to measure network latency differences only.
Explanation:An A/B test for models randomly assigns real traffic to two (or more) model variants and compares a defined outcome metric between groups to decide which performs better in production. (a) describes a determinism check, unrelated to comparing variants. (b) describes offline train/validation evaluation. (d) describes a latency benchmark, not a model comparison.
Aimlops Deployment Strategies MlDifficulty 2
What is a 'rollback trigger' in the context of ML model deployment?
- aA manual ticket a developer files days after a deployment to request a future code review.
- bA predefined condition (e.g., a monitored metric crossing a threshold) that causes the serving system to automatically or semi-automatically revert to the last known-good model version.✓
- cA configuration flag that permanently disables all future deployments once set.
- dA scheduled nightly job that retrains the model from scratch regardless of current performance, with no connection to any monitored metric or threshold.
Explanation:A rollback trigger ties a concrete, monitored condition (error rate spike, latency regression, drift alert, etc.) to an automatic or approval-gated revert action, so degraded behavior doesn't linger. (a) describes an unrelated manual process with no automation. (c) and (d) describe unrelated mechanisms that don't revert a bad deployment.
Aimlops Deployment Strategies MlDifficulty 1
What does 'multi-model serving' (or model routing) mean?
- aMultiple model versions run concurrently behind one serving layer, and a router directs each incoming request to the appropriate version based on rules like traffic weight, user segment, or request type.✓
- bOnly one model version can ever exist in production at a time, and any new version must fully replace it before serving resumes.
- cA single model is copied to multiple identical replicas purely to share load evenly, with no version differences involved.
- dTraining multiple models simultaneously on the same GPU to reduce total training time.
Explanation:Multi-model serving/routing is specifically about serving several distinct model versions (or variants) at once and routing requests among them based on defined rules. (b) describes single-version-only serving, the opposite. (c) describes plain load-balanced replicas of one identical model, no routing between versions. (d) describes a training-time concept, unrelated to serving.