Sample questions
Aimlops Experiment Tracking ReproducibilityDifficulty 1
What is the primary purpose of an experiment tracking system in ML development?
- aTo record each training run's configuration, metrics, and artifacts so past results can be compared and understood later.✓
- bTo replace source code version control entirely.
- cTo automatically select the best hyperparameters without any human review, replacing the need for a data scientist to interpret results.
- dTo serve trained models to production traffic in real time.
Explanation:Experiment tracking exists to capture what was run, with what settings, and what happened, so results remain comparable and explainable over time. (c) describes automated hyperparameter search, a related but separate concept. (b) confuses it with git/version control, which it complements rather than replaces. (d) describes model serving, an unrelated production concern.
Aimlops Experiment Tracking ReproducibilityDifficulty 1
What does 'run metadata' in an experiment tracking system typically include, at minimum?
- aA screenshot of the terminal window at the moment training finished, kept as the primary record instead of any structured, queryable data.
- bThe physical location of the GPU rack the job happened to run on, tracked so the hardware team can plan future datacenter capacity.
- cHyperparameters, key metrics, and enough context (e.g. code version, dataset reference) to identify what produced a given result.✓
- dOnly the final accuracy number, since intermediate values are not useful.
Explanation:Useful run metadata captures the configuration and context needed to identify and later reproduce or compare a result: hyperparameters, metrics, and references to code/data. (d) discards the intermediate history that's often needed to diagnose training behavior. (b) and (a) describe incidental, non-actionable details that don't help identify what produced a result.
Aimlops Experiment Tracking ReproducibilityDifficulty 1
Why is it useful to log hyperparameters alongside metrics for each run?
- aA performance benchmark of the current environment's numpy and torch installations, run automatically before every training job starts.
- bBecause hyperparameters change the training framework's license terms.
- cSo the training job can be billed correctly to the right department.
- dSo later you can tell which configuration produced a given metric value, instead of relying on memory.✓
Explanation:Logging hyperparameters next to metrics ties a result to the exact settings that produced it, avoiding guesswork weeks or months later. (b) and (c) invent licensing/billing effects unrelated to tracking. (a) is false; loading a saved model doesn't require an external metadata log, though having one helps interpret it.
Aimlops Experiment Tracking ReproducibilityDifficulty 2
What is the benefit of logging a metric (e.g. validation loss) at every epoch or step, rather than only logging the final value?
- aIt lets you see the full training curve (e.g. spot overfitting or instability), not just the end result.✓
- bEvery experiment tracking tool's API requires it and refuses final-only logging, regardless of which framework or language is used.
- cIt reduces the total storage used by the experiment tracker.
- dIt automatically fixes any bug present in the training loop.
Explanation:A per-step/epoch curve reveals patterns — overfitting, plateaus, instability — that a single final number hides. (c) is backwards; logging more points uses more, not less, storage. (d) fabricates an automatic bug-fixing effect logging doesn't have. (b) invents a universal API requirement that doesn't exist across tools.
Aimlops Experiment Tracking ReproducibilityDifficulty 2
Why record the exact git commit hash (and whether the working tree was 'dirty', i.e. had uncommitted changes) together with a training run?
- aBecause pip requires a git hash before it will install any package, even ones with no dependency on the current repository.
- bBecause git commit hashes are needed to compute the model's accuracy.
- cSo the exact code state that produced the run can be identified later, even if the branch has since moved on.✓
- dSo the experiment tracker can automatically merge conflicting branches.
Explanation:The commit hash (plus a dirty-tree flag) pins down exactly which code, including any uncommitted edits, produced a run, which matters once the branch has moved past that point. (b) confuses code identity with the training math itself. (a) fabricates a pip dependency on git hashes. (d) invents an auto-merge capability trackers don't have.
Aimlops Experiment Tracking ReproducibilityDifficulty 2
Why is dataset versioning (not just code versioning) important for reproducibility?
- aBecause datasets never change once created, so versioning them is only a formality that experienced teams can safely skip entirely.
- bBecause dataset versioning replaces the need for logging hyperparameters.
- cBecause only code changes affect model output, not the underlying data, so a team can safely ignore which dataset snapshot was actually used.
- dBecause the same code trained on a different dataset snapshot can produce different results, so the data version is part of what defines a run.✓
Explanation:Data drifts, gets corrected, or is re-collected over time, so which data snapshot was used is as much a part of a run's identity as the code was. (a) is false; datasets do get updated, corrected, or extended in practice. (b) and (c) wrongly claim one tracked dimension substitutes for or excludes another, when code, data, and hyperparameters are all independently relevant.