Sample questions
Ml Experimentation ReproducibilityDifficulty 1
Which of the following is generally considered essential metadata to log for an experiment, so its result can be reproduced later?
- aOnly the final accuracy number, since that's what matters most
- bThe output visualizations, since they show what happened
- cThe name of the person who happened to run the experiment
- dThe code version, data version, and hyperparameter values used✓
Explanation:Reproducing a result requires knowing exactly what produced it: which code, which data, and which settings (d). A final number alone (a) can't be traced back to its cause. Visualizations (b) describe an outcome, not the configuration behind it. Who ran it (c) is irrelevant to whether the run itself can be recreated.
Ml Experimentation ReproducibilityDifficulty 1
Why can the exact same training code produce a somewhat different model when run twice on the same dataset?
- aRandomness sources like initialization or shuffling differ between runs unless a seed is fixed✓
- bThe dataset silently changes its meaning between the two runs
- cModel training is deterministic by default in every framework, so this should never actually happen
- dThe metric calculation itself always changes between runs
Explanation:Unless randomness is pinned with a seed, things like weight initialization or data shuffling order can differ run to run, changing the outcome (a). The dataset's meaning (b) doesn't change on its own. Training is not deterministic by default in most setups (c is wrong). Metric calculation code is fixed and doesn't randomly change (d is wrong).
Ml Experimentation ReproducibilityDifficulty 1
What does 'fixing a random seed' aim to achieve for an experiment?
- aIt removes the need for a validation set entirely
- bIt guarantees identical results on completely different hardware
- cIt makes random operations like shuffling and initialization follow the same sequence on each run✓
- dIt permanently prevents the model from overfitting
Explanation:A fixed seed forces pseudo-random operations to produce the same sequence each run, which improves (though doesn't fully guarantee) repeatability (c). It has nothing to do with validation sets (a) or overfitting (d). It also doesn't guarantee identical results across different hardware, since hardware-level numeric behavior can still differ (b is wrong).
Ml Experimentation ReproducibilityDifficulty 1
A junior engineer reruns a training script a week later using what looks like the same code file, but gets a noticeably different result. What is the most likely basic explanation to check first?
- aThe programming language itself has changed its arithmetic rules
- bThe data, library versions, or seed may differ from before, even though the code file looks the same✓
- cMachine learning models are inherently required to give a different result every single time by design
- dThe metric used must have been calculated incorrectly the first time
Explanation:A code file 'looking the same' doesn't guarantee the data, library versions, or seed behind it are unchanged — any of those can drift silently (b). Languages don't quietly change arithmetic rules (a). ML training isn't required to be non-reproducible by design (c). Jumping to 'the first metric was wrong' (d) skips checking the more likely causes first.
Ml Experimentation ReproducibilityDifficulty 1
Which of these is a basic reason full reproducibility of an ML experiment is harder than reproducing a typical deterministic software build?
- aML code cannot be version controlled the same way regular code can
- bMetrics like accuracy are never comparable between two separate runs
- cProgramming languages do not support fixing random seeds
- dResults depend on data snapshots, seeds, and environment details, any of which can silently drift✓
Explanation:Beyond code, an ML result also depends on the data snapshot, seed, and environment — several extra moving parts that a plain software build doesn't have (d). ML code is version-controlled the same way as any other code (a is wrong). Metrics remain comparable when the setup is controlled (b is wrong). Fixing seeds is a common, supported practice (c is wrong).
Ml Experimentation ReproducibilityDifficulty 1
In the context of experiment tracking, what is meant by 'data version'?
- aThe programming language version used to write the training script
- bThe exact dataset snapshot used to produce a given result✓
- cThe number of times the dataset file has been renamed
- dThe folder path where the dataset happens to be stored
Explanation:Data version identifies exactly which snapshot of the data was used, so the same data can be used again later (b). It has nothing to do with the language version (a), rename count (c), or a storage path (d) — a path can point to different content over time, which is precisely the problem versioning solves.