Sample questions
Ti Input Pipeline ThroughputDifficulty 1
A dataset is stored as 100 shard files. A training job runs on 8 ranks, and each rank is assigned a contiguous slice of the shard list at startup. Since 100 is not evenly divisible by 8, what do you observe about how many shards each rank ends up with?
- aMost ranks get 12 shards and a few ranks get 13 shards; the shard count differs by at most one between ranks, since the leftover shards are handed out one per rank.✓
- bAll 8 ranks get exactly 12 shards, and the remaining 4 shards are simply never read by anyone that epoch.
- cThe job refuses to start at all, because the shard framework treats an uneven split as a fatal configuration error.
- dOne rank ends up reading all 100 shards on behalf of the others, then broadcasts the decoded samples over the network.
Explanation:A standard contiguous split hands out 100 // 8 = 12 shards to every rank and then distributes the 100 % 8 = 4 leftover shards one each to the first four ranks, so counts land at either 12 or 13 — never a difference of more than one shard. Nothing forces the job to abort on an uneven split, no shard is silently skipped, and there is no single-rank read-and-broadcast step.
Ti Input Pipeline ThroughputDifficulty 2
A team splits 17 shard files contiguously across 4 ranks. Rank 0 ends up with 5 shards while ranks 1, 2, and 3 each get 4 shards. If shard sizes are roughly equal, what does this most directly cause?
- aRank 0 runs more local steps that epoch than the others.✓
- bNothing measurable happens in practice, because a one-shard gap between 5 and 4 is far too small to ever change how many local steps any rank takes, regardless of shard size.
- cRank 0 crashes on startup, since receiving a shard count that differs from its peers is treated as an invalid assignment by most shard-based loaders.
- dThe extra shard assigned to rank 0 is automatically detected and silently reassigned mid-epoch to whichever rank finishes its local data first, keeping every rank's step count equal.
Explanation:With one extra shard's worth of samples, rank 0 has strictly more local data than ranks 1-3; iterating each rank's local dataset to exhaustion means rank 0 takes more steps before it runs out. The gap is real (not negligible by default), nothing crashes on a mismatched count, and nothing silently rebalances a shard mid-epoch.
Ti Input Pipeline ThroughputDifficulty 2
To avoid giving ranks unequal numbers of shards when the shard count does not divide evenly by the rank count, one common fix is to pad the shard list: repeat a few shards (typically taken from the start of the list) until the total is divisible by the rank count. What is the direct consequence of this padding, for the ranks that receive a repeated shard?
- aThey read that shard's samples an extra time during the epoch, so a small number of samples in the dataset end up seen more than once.✓
- bThey read the padded shard once, exactly like every other shard, since padding only changes which rank owns a shard, not how many times it is read.
- cThey skip that shard entirely, since padded (repeated) shards are marked and excluded from the actual read to keep sample counts exact.
- dThey read a placeholder of empty (all-zero) samples in place of the repeated shard, so real data is not duplicated.
Explanation:Padding works by literally duplicating some shard entries in the assignment list so the total count divides evenly; whichever ranks receive those duplicated entries read that shard's samples an extra time in that epoch. It is not read only once like a normal shard, it is not skipped, and it is not replaced by empty placeholder data — real samples are actually repeated.
Ti Input Pipeline ThroughputDifficulty 1
A streaming data loader keeps a fixed-size shuffle buffer: it fills the buffer with the next N samples from the stream, then each time it emits a sample it randomly picks one from the buffer and replaces it with the next unread sample. Compared to a true global shuffle of the whole dataset, what does this buffer-based approach give you?
- aExactly the same output order as a global shuffle, as long as N is at least 2, because any nonzero buffer size is mathematically equivalent to shuffling the whole stream at once.
- bA completely sequential, unshuffled order, because the buffer only holds a fixed lookahead window and drains items in the same front-to-back order it filled them, so nothing inside that window gets reordered.
- cAn approximate, local shuffle: a sample can only move to a position within roughly N of its original one, which stays weaker than a true global shuffle overall.✓
- dA shuffle that reorders samples only within the very first buffer-full and then leaves the rest of the stream completely untouched in its original order.
Explanation:Because a sample can only be swapped into the buffer and later emitted while it sits among at most N buffered items, its output position is bounded by roughly the buffer size relative to its original position — a local, approximate shuffle, not a true global one. It is not equivalent to a global shuffle regardless of N, it is not literally unshuffled, and reordering continues throughout the stream (within that bounded window), not only in the first bufferful.
Ti Input Pipeline ThroughputDifficulty 2
A dataset's shard files were written sequentially by class label: the first shards contain only class 0, the next shards only class 1, and so on. A shuffle buffer holding 200 samples is used, and each class block is 1000 samples. What do you expect batches drawn early within a class block to look like?
- aBalanced enough across classes that the class-sorted file layout is no longer visible in any batch, because a 200-sample buffer already mixes in enough of the next block to hide the original ordering.
- bHeavily dominated by the class currently being read.✓
- cDominated by whichever class happens to appear last in the overall file order, no matter which block is currently being read from.
- dStatistically identical to batches drawn from a true global shuffle of the same 10,000-sample dataset, since the buffer, however small, still removes any dependence on write order.
Explanation:A buffer of 200 sitting inside a 1000-sample same-class block is filled almost entirely with that one class, so batches drawn from it stay heavily skewed toward that class until the buffer starts refilling from the next block. A buffer this much smaller than the block cannot hide the sequential-by-class layout, it is not dominated by an unrelated 'last' class, and it clearly differs from a true global shuffle.
Ti Input Pipeline ThroughputDifficulty 3
# stream is sorted by class: [0]*1000 + [1]*1000 + ... + [9]*1000
def batch_majority_purity(shuffled, batch_size=32):
# fraction of the batch belonging to its most common class
...
Measuring
batch_majority_purity across a full epoch for a class-sorted stream with buffer size equal to one class block (1000), the average purity comes out around 0.55, while a true global shuffle of the same data gives an average purity around 0.19. What does this measured gap tell you?
- aThe measurement must be wrong: any shuffle buffer at least as large as one class block should fully match a global shuffle's purity.
- bThe gap only exists for buffer sizes smaller than 1000; a buffer of exactly 1000 samples should already match the global-shuffle number.
- cEven a buffer as large as one full class block still leaves batches noticeably more class-skewed than a global shuffle, confirming buffer size alone does not guarantee shuffle quality.✓
- dThe 0.19 global-shuffle number is itself unreliable and biased, so no real conclusion can be drawn by comparing it against 0.55.
Explanation:A buffer sized to exactly one class block is still mostly refilled from within that same block much of the time, so 0.55 sits well above the 0.19 global-shuffle baseline rather than matching it — reaching baseline-level purity needs a buffer that is a multiple of the block size, not merely equal to it. Nothing here indicates a measurement error or an unreliable baseline.