Sample questions
Ti Training Determinism NumericsDifficulty 3
a = torch.tensor(1.0e16, dtype=torch.float32)
b = torch.tensor(1.0, dtype=torch.float32)
c = torch.tensor(-1.0e16, dtype=torch.float32)
print(((a + b) + c).item())
print((a + (b + c)).item())
print(((a + c) + b).item())
This prints
0.0,
0.0, then
1.0. All three lines add the same three numbers. What explains the third result differing from the first two?
- aA print-statement bug swapped which variable each line displays
- b
torch.tensor cached the first two calls and returned a stale value - cThe grouping order changed, and float32 addition isn't associative✓
- dFloat32 addition is undefined whenever a negative operand appears
Explanation:(a + c) cancels the two huge-magnitude values to 0.0 first, so the remaining + b leaves exactly 1.0. In the first two groupings, 1.0 is combined with 1e16 before the cancellation — and 1e16 in float32 can't represent an increment of 1.0, so it rounds away (c). There's no print bug (a), no caching in torch.tensor (b), and negative operands are perfectly well-defined in IEEE 754 (d).
Ti Training Determinism NumericsDifficulty 1
A trainee reruns the exact same single-GPU training step twice with an identical seed, but changes how the per-sample losses in a batch are summed (left-to-right loop vs. a tree-shaped pairwise sum). The two totals differ in the last couple of decimal digits. What is the most accurate description?
- aAn expected effect, since floating-point addition is not associative✓
- bA sign the random seed wasn't actually applied on one run
- cOnly possible if one summation method has an indexing bug
- dNot expected unless the two runs used different hardware
Explanation:Floating-point addition is not associative: (x+y)+z and x+(y+z) can round differently even though both are mathematically the same sum. Changing the summation strategy changes the grouping, so a small last-digit difference is expected, not a bug (a). The seed governs randomness, not deterministic summation order (b). No indexing bug is required (c), and this has nothing to do with hardware differences between the two runs (d).
Ti Training Determinism NumericsDifficulty 3
A team reruns the exact same training script — identical code, data, and a fixed seed — on two machines that differ only in which minor version of PyTorch happens to be installed (a routine dependency update landed on one of them). The final loss differs starting from step 1 by a tiny amount. Is this necessarily a sign that the seed wasn't fixed correctly?
- aNo — a fixed seed pins randomness only, not which kernel a library version ships✓
- bYes — a fixed seed guarantees bit-identical results regardless of which library version is installed
- cNo — that would require PyTorch to silently disable seeding whenever it detects a version mismatch, which is not what happens
- dYes, but only if the two machines also run different operating systems
Explanation:A fixed seed controls the sequence of 'random' numbers produced, but says nothing about which specific kernel implementation or default algorithm a given library version ships — a routine version bump can change a default kernel's internal grouping or algorithm choice, producing a tiny but genuine rounding difference even with byte-identical code, data, and seed (a). The seed doesn't extend any guarantee across library versions (b). Nothing about a version difference disables seeding (c), and this isn't specifically an operating-system issue (d).
Ti Training Determinism NumericsDifficulty 1
In a 4-rank data-parallel job, every rank calls torch.manual_seed(42) right before building its augmentation pipeline. Each rank still sees a different shard of images. What is the observable consequence?
- aAugmentation differs across ranks anyway, since DataLoader ignores this seed
- bEvery rank's augmentation stream produces the same sequence of decisions✓
- cThe job crashes since the seed value is identical on more than one rank
- dRandom augmentation is disabled once an identical seed is detected twice
Explanation:Seeding every rank with the same value puts every rank's RNG generator into the identical internal state, so the k-th draw from that generator is the same number on every rank; whatever augmentation decision that number maps to therefore lands at the same position in each rank's pipeline, just applied to that rank's own images (b). Nothing here disables augmentation or crashes the job (c, d), and DataLoader does respect a seed set this way (a).
Ti Training Determinism NumericsDifficulty 2
t = torch.tensor([1.0, 3.0, 3.0, 2.0, 3.0])
print(t.argmax().item())
The maximum
3.0 appears at indices 1, 2, and 4. What does this print?
- a4, the last index holding the maximum value
- bA different index chosen at random on each run
- cA RuntimeError, since the maximum isn't unique
- d1, the first index holding the maximum value✓
Explanation:torch.argmax() deterministically returns the first index at which the maximum occurs — here index 1 (d), not the last (a) and not a random pick (b). It runs without error on ties (c).
Ti Training Determinism NumericsDifficulty 2
Two runs of the same GPU step, same inputs, same seed, use a gradient-scatter kernel that accumulates contributions from many threads with atomic add instructions. The resulting gradients differ at the last few mantissa bits. What is the most direct cause?
- aThe GPU's memory cache silently retains stale gradient values left over between the two runs
- bThe randomness comes from the CPU host code, not the GPU kernel
- cAtomic-add accumulation doesn't fix arrival order✓
- dThis can only happen on an unsupported PyTorch installation
Explanation:Atomic-add kernels let threads add into a shared accumulator whenever they finish, and thread completion order isn't fixed between runs — so the summation order, and hence the exact rounded float result, can vary run to run even with identical inputs (c). No stale-value caching is involved (a); the source is the GPU kernel's execution order, not host-side code (b); and this is a documented property, not corruption (d).