Sample questions
Ti Distributed Strategies SyncDifficulty 3
You run ring all-reduce to sum gradients across 8 ranks. Each rank's local gradient tensor is 800 MB. Roughly how much data does EACH rank send over the network in total during the ring all-reduce, ignoring protocol overhead?
- a800 MB, since the tensor is forwarded just once per rank.
- b6400 MB, a full copy received from all 7 other ranks.
- cAbout 1400 MB, since the 2(N-1)/N factor applies here.✓
- d100 MB: no chunk is ever re-sent.
Explanation:Ring all-reduce moves the tensor around the ring in chunk-sized pieces over two phases, so each rank ends up sending and receiving roughly (N-1)/N of the tensor twice, not a handful of full copies; total traffic per rank is about 2(N-1)/N size. With N=8 and 800 MB that is 27/8*800 ≈ 1400 MB — far less than sending 7 full copies.
Ti Distributed Strategies SyncDifficulty 1
In ring all-reduce with N ranks, how many total communication steps (reduce-scatter + all-gather) does the algorithm run, independent of tensor size?
- aN steps, one per rank.
- b2(N-1) steps: reduce-scatter, then all-gather.✓
- clog2(N) steps, like a tree reduction.
- d1 step, a single collective call.
Explanation:Ring all-reduce is built from two phases, each requiring N-1 point-to-point exchanges around the ring: a reduce-scatter phase that leaves each rank with one reduced chunk, then an all-gather phase that distributes every chunk to every rank. That is 2(N-1) steps total, not log2(N) — the log2(N) step count belongs to tree/halving-doubling style reductions instead.
Ti Distributed Strategies SyncDifficulty 2
Two ranks run this DDP training step with gloo backend (PyTorch 2.8):
torch.manual_seed(0)
m = nn.Linear(2, 1, bias=False)
ddp = DDP(m)
ddp(torch.full((1, 2), float(rank + 1))).sum().backward()
Rank 0's local (unsynced) gradient would be [1, 1] and rank 1's would be [2, 2]. What does
m.weight.grad actually hold on both ranks after
backward() returns?
- a[1.5, 1.5] on both ranks, the averaged value.✓
- b[3.0, 3.0] on both ranks, the summed value.
- c[1.0, 1.0] on rank 0 and [2.0, 2.0] on rank 1, unsynced.
- d[1.5, 1.5] on rank 0 only; rank 1 keeps [2.0, 2.0].
Explanation:This was measured directly: dist.all_reduce(t, op=SUM) on the same values gives 3.0, but DDP's built-in backward hook divides the summed gradient by world_size before writing it into .grad, so both ranks end up with the average [1.5, 1.5], not the sum, and both ranks see the identical value after backward().
Ti Distributed Strategies SyncDifficulty 3
A team migrates from single-GPU training to 4-rank DDP. To keep the code 'simple', they replace DDP's comm hook with a custom one that calls dist.all_reduce(bucket.buffer(), op=SUM) directly, without dividing by world size (this was tested: doing so leaves .grad at the raw cross-rank SUM, not the average). Keeping the same learning rate as the single-GPU run, what is the most likely immediate symptom?
- aNo change; PyTorch auto-rescales the optimizer's LR.
- bTraining becomes bit-deterministic across runs.
- cThe loss curve is identical to the single-GPU run.
- dEffective step size is roughly 4x too large for the LR.✓
Explanation:An update is w -= lr * grad. If grad is now the sum across 4 ranks instead of the average, it is roughly 4x the magnitude a correctly-scaled gradient would have (for similar per-rank gradients), so the effective step size scales up by about world_size — commonly seen as instability or divergence unless the learning rate is divided by world_size to compensate.
Ti Distributed Strategies SyncDifficulty 2
Rank 0 processes a local batch of 1 sample and rank 1 processes a local batch of 3 samples in the same DDP step (an uneven split). This was measured: DDP's synced gradient equals the FLAT average of each rank's local per-sample-mean gradient (1/world_size), not a sample-count-weighted average across all 4 samples combined. What does this imply?
- aNothing in practice; DDP re-weights by local batch size first.
- bThe synced gradient is undefined (NaN) when batch sizes differ.
- cRank 0's sample is silently dropped from the average.
- dRank 1's 3 samples are under-weighted vs a true pooled mean.✓
Explanation:A true pooled mean over the 1+3=4 samples would weight rank 1's contribution 3x more than rank 0's. But DDP's flat 1/world_size average gives rank 0's local mean (over 1 sample) the same weight as rank 1's local mean (over 3 samples), so rank 1's samples end up under-represented relative to a sample-count-weighted mean.
Ti Distributed Strategies SyncDifficulty 2
In DDP, gradient 'bucketing' groups multiple parameters' gradients into one buffer before running a single all-reduce over that buffer. What is the main reason DDP does this instead of running one all-reduce per parameter?
- aIt reduces per-rank GPU and host memory footprint, since bucketed gradients replace the need to keep separate per-parameter communication buffers alive.
- bIt's mathematically required; naive per-parameter averaging is not numerically valid without first grouping tensors.
- cIt cuts per-call overhead and overlaps with backward.✓
- dIt is intended to make results bit-identical across backends.
Explanation:Launching a separate collective per parameter would mean hundreds of tiny, high-overhead calls. Bucketing batches gradients into fewer, larger all-reduce calls, and — because backward computes gradients layer by layer — a bucket that becomes fully ready can start communicating while backward keeps computing gradients for other layers, overlapping communication with compute.