Sample questions
Aipy Gpu Memory ManagementDifficulty 1
When a PyTorch training script raises CUDA out of memory, what does this error most directly indicate?
- aThe Python process ran out of regular (host) RAM, not GPU memory.
- bThe GPU could not allocate the requested block of device memory for a tensor or operation.✓
- cThe CUDA driver version is incompatible with the installed PyTorch build. This can sound plausible since driver mismatches are also reported at startup, but the wording and context differ from an allocation failure.
- dThe model's weights file on disk is corrupted and failed to load.
Explanation:CUDA out of memory is raised when the CUDA memory allocator cannot satisfy a request for GPU device memory, because the GPU's memory is already occupied by other tensors (weights, activations, gradients, optimizer state, cached blocks). (a) describes a host-RAM error, which raises a different exception. (c) and (d) describe unrelated failure modes (driver mismatch, corrupted checkpoint) that produce different error messages.
Aipy Gpu Memory ManagementDifficulty 2
A team trains an image classifier and hits CUDA out of memory only when they increase batch_size from 32 to 128, with everything else unchanged. What is the most likely reason?
- aActivation memory scales roughly linearly with batch size, so a 4x larger batch needs roughly 4x more activation memory.✓
- bLarger batch sizes force PyTorch to store the model weights in a higher-precision format. Teams sometimes reach for this explanation because precision and memory both sound related to 'how big' a tensor is.
- cThe optimizer creates a separate copy of the model for every sample in the batch.
- dBatch size only affects how many gradient updates occur per epoch, not memory usage.
Explanation:For most architectures, per-layer activations are stored per sample in the batch, so total activation memory grows roughly linearly with batch size, while the memory used by the model weights themselves stays constant. (b) and (c) invent mechanisms that don't reflect how batch size affects memory. (d) is wrong because batch size directly affects memory pressure, not just update frequency.
Aipy Gpu Memory ManagementDifficulty 2
After a CUDA out of memory error, a developer calls torch.cuda.empty_cache() hoping to fix the underlying problem and retries the same batch size. What actually happens?
- aIt permanently increases the total GPU memory available to the process.
- bIt frees the memory still held by tensors that are currently referenced by Python variables. This would actually be dangerous if true, since it would silently corrupt a running computation, which is not what this call does.
- cIt returns currently unused, cached memory blocks back to the CUDA driver, but does not free memory still in use, so the same OOM can recur.✓
- dIt compresses all tensors currently on the GPU to reduce their memory footprint.
Explanation:empty_cache() only releases memory blocks that PyTorch's caching allocator is holding but not currently using back to the CUDA driver; it cannot touch memory still referenced by live tensors, and it does not add new physical memory. If the batch genuinely needs more memory than the GPU has, the same error will happen again. (a) and (d) describe capabilities this call does not have. (b) misdescribes it as freeing live, referenced memory, which would break correctness.
Aipy Gpu Memory ManagementDifficulty 1
Training a model in fp16 or bf16 mixed precision instead of full fp32, roughly how does the memory needed to store weight and activation tensors change?
- aIt is roughly halved, since each value uses 2 bytes instead of 4.✓
- bIt roughly doubles, since two values are now needed to represent each original number.
- cIt stays exactly the same, since precision only affects numerical accuracy, not storage size.
- dIt depends only on batch size, not on the numeric format used. This sounds intuitive if you assume memory tracking is tied to computation speed rather than to the numeric format's byte width.
Explanation:fp16 and bf16 both use 2 bytes per value versus fp32's 4 bytes, so tensors stored in these formats roughly halve their memory footprint compared to the same tensor in fp32. (b) inverts the direction. (c) and (d) deny that numeric format affects storage size, which contradicts how these dtypes are defined.
Aipy Gpu Memory ManagementDifficulty 2
Gradient checkpointing is enabled on a model that was previously running out of GPU memory during the backward pass. What trade-off does this technique make?
- aIt reduces the number of training steps needed, at the cost of lower final accuracy.
- bIt reduces memory by permanently deleting a fraction of the model's layers. This might seem plausible if pruning is confused with a memory-saving technique, but checkpointing never removes layers.
- cIt reduces memory by storing all activations in host RAM instead of GPU memory.
- dIt reduces memory by not storing most intermediate activations, recomputing them during the backward pass instead, at the cost of extra compute time.✓
Explanation:Gradient checkpointing avoids storing most forward-pass activations; during backpropagation, it recomputes those activations from the last stored checkpoint, trading additional forward-pass compute for lower peak memory. (a) and (b) describe unrelated or destructive changes checkpointing does not make. (c) confuses it with CPU/activation offloading, a different technique.
Aipy Gpu Memory ManagementDifficulty 2
In the context of GPU memory, what does 'memory fragmentation' refer to?
- aThe GPU's total memory capacity being physically divided across multiple chips. Physical chip layout is a real hardware detail, but it is unrelated to how the allocator tracks free and used regions of memory.
- bFree memory existing in many small, non-contiguous blocks, so a large-enough single allocation can fail even though the total free memory is sufficient.✓
- cA model's weights being split into shards for distributed training.
- dThe gradual corruption of tensor values due to repeated fp16 rounding.
Explanation:Fragmentation happens when memory frees and allocates over time in a way that leaves free space scattered in small non-contiguous chunks; even if their sum is large enough, no single chunk may be big enough for a new large allocation. (a) and (c) describe unrelated hardware/parallelism concepts. (d) describes numerical precision drift, not a memory layout issue.