Sample questions
Ml Scaling PerformanceDifficulty 1
In distributed training, what does 'data parallelism' mean at a basic level?
- aSplitting a single model's layers across multiple devices.
- bRunning unrelated models on different machines and averaging their predictions.
- cStoring training data in one central database queried record by record.
- dGiving each worker a full model copy and a data slice, then merging gradients.✓
Explanation:Data parallelism replicates the full model on every worker while splitting the dataset across workers; each worker computes gradients on its own data slice, and these gradients are combined (e.g. averaged) to update the shared model. This is different from splitting the model itself across devices.
Ml Scaling PerformanceDifficulty 1
What does 'model parallelism' mean, and when is it typically needed?
- aCopying the whole model onto every worker to run predictions in parallel.
- bSplitting the training data into random shuffled batches before each epoch.
- cTraining several unrelated small models and picking the best one afterward.
- dSplitting model parameters across devices when too large for one device.✓
Explanation:Model parallelism partitions a single model's parameters (e.g. across layers) across multiple devices, so pieces of one forward/backward pass run on different devices. It is used when a model does not fit into the memory of a single device, unlike data parallelism where each device holds a full model copy.
Ml Scaling PerformanceDifficulty 1
In the context of model serving, what is the basic difference between throughput and latency?
- aThroughput is requests processed per unit time; latency is time per single request.✓
- bThroughput is time per single request; latency is requests processed per unit time.
- cThroughput and latency measure the same thing in different units.
- dThroughput applies only to training jobs; latency only to storage systems.
Explanation:Throughput measures the rate of work completed (e.g. requests per second), while latency measures the time a single request takes from submission to response. Optimizing one does not automatically optimize the other, and the two are often in tension (see batching).
Ml Scaling PerformanceDifficulty 1
In a model serving system, what does 'caching' basically achieve?
- aIt permanently increases prediction accuracy for every input.
- bIt removes the need for any monitoring of the serving system.
- cIt stores past results so repeated requests skip recomputation.✓
- dIt guarantees instant responses even for entirely novel requests.
Explanation:Caching stores results of prior computations (e.g. predictions for a given input) so that when the same or a similar request arrives again, the stored result can be reused instead of running the full computation again. It helps with repeated inputs but does not speed up genuinely novel requests, and it has no effect on model accuracy.
Ml Scaling PerformanceDifficulty 1
What is a 'straggler' in the context of synchronous distributed training?
- aA worker that finishes early and is removed from the job.
- bA batch of training data duplicated across two workers by mistake.
- cA checkpoint that fails to load correctly after a crash.
- dA worker much slower than the others, delaying that step's update.✓
Explanation:In synchronous distributed training, all workers must finish a step before the shared update happens. A straggler is a worker that is unusually slow on a given step (e.g. due to a slow machine or uneven data), so every other worker sits idle waiting for it, reducing overall training speed.
Ml Scaling PerformanceDifficulty 1
A team notices that when they process inference requests one at a time, their accelerator (GPU/similar hardware) sits mostly idle between requests, and overall requests-per-second is low. What is the most basic explanation for why grouping several requests into a batch would help?
- aIt makes each individual prediction more accurate.
- bIt removes the need to ever retrain the model.
- cIt eliminates network latency between client and server.
- dIt keeps the accelerator busy instead of sitting idle.✓
Explanation:Processing one request at a time under-uses hardware that is capable of parallel computation, since it spends time idle between requests. Grouping requests into a batch lets the hardware do more useful work per unit of time, increasing throughput — though it can add some latency per individual request while waiting for a batch to fill.