Sample questions
Ti Cluster Scheduling ResourcesDifficulty 1
A training job requests 8 GPUs with gang (all-or-nothing) scheduling: either all 8 ranks start together or none start. The cluster currently has 8 GPUs free in total, but they sit on 8 different nodes with only 1 free GPU per node, while the job's launch script requires all 8 GPUs on a single node. What happens to the job?
- aIt starts immediately, using one free GPU per node across the 8 nodes, since the scheduler only tallies the cluster-wide total.
- bIt starts with 7 GPUs now and adds the 8th once available.
- cThe scheduler cancels the job outright for lacking any pre-labeled, GPU-tagged nodes to place it on.
- dIt stays pending; no single node has 8 free GPUs for its single-node requirement.✓
Explanation:Gang scheduling checks the job's actual placement constraint, not just the cluster-wide free count; 8 GPUs one-per-node cannot host an 8-GPU single-node job.
Ti Cluster Scheduling ResourcesDifficulty 3
A cluster has 10 nodes of 8 GPUs each (80 total). At a given moment, 16 GPUs are free, but spread as exactly 2 free GPUs on each of 8 different nodes. A pending job needs 8 GPUs on a single node (its ranks use a fast intra-node link and cannot span nodes). Later, running jobs finish and the same 16 free GPUs end up concentrated as 8 free GPUs each on only 2 nodes. What changed for the pending job?
- aNothing changed; the free GPU count stayed at 16 both times.
- bIt still cannot start; 2 nodes cannot host a job spread across several nodes, regardless of how much capacity sits on each one.
- cIt can now start, since the same idle capacity now sits on nodes large enough.✓
- dThe scheduler must restart every running job to free up GPUs first.
Explanation:Placement, not just total free count, decides whether a single-node job can start: 16 GPUs went from unusable, scattered, to sufficient, concentrated, purely by arrangement.
Ti Cluster Scheduling ResourcesDifficulty 2
A cluster's gang scheduler admits jobs as groups defined by a spec like the one below.
apiVersion: scheduling.k8s.io/v1
kind: PodGroup
spec:
minMember: 8
What does
minMember: 8 enforce here?
- aIt starts pods one at a time until 8 have launched.
- bThe group starts only once at least 8 pods can be placed at the same time.✓
- cThe job is limited to a maximum of 8 pods for its lifetime.
- dThe scheduler reserves 8 extra idle pods as a buffer.
Explanation:minMember is the gang-scheduling threshold for simultaneous admission, not a start-order rule, a lifetime cap, or an idle-buffer reservation.
Ti Cluster Scheduling ResourcesDifficulty 2
A queue processes jobs first-come-first-served, but resources free up gradually as small jobs finish. A large job needing 32 GPUs has sat at the head of the queue for hours without starting, because a steady stream of newly submitted 4-GPU jobs keeps grabbing GPUs the moment they free up. What is this pattern called, and what feature directly addresses it?
- aThis is starvation; a reservation guarantees the large job's turn once GPUs accumulate.✓
- bThis is priority inversion; the fix is equal priority for all jobs, since that gap is what created the imbalance.
- cThis is thrashing; the fix is reducing the cluster's node count.
- dThis is deadlock; the fix is resubmitting the job with fewer GPUs.
Explanation:The large job isn't blocked by a circular dependency or resource oscillation, it simply never gets its turn, which a guaranteed reservation directly prevents.
Ti Cluster Scheduling ResourcesDifficulty 3
A scheduler uses backfill: a smaller job may run ahead of a larger job queued earlier, but only if doing so does not delay the larger job's earliest possible start time. The large job (queued first, needs 16 GPUs) is projected to have enough GPUs to start in 3 hours. A newly submitted small job needs 2 currently-free GPUs and is estimated to run for 2 hours. Does backfill allow the small job to run now?
- aNo, backfill only ever runs jobs strictly in submission order.
- bNo, backfill applies only to jobs of the same size class.
- cYes; its 2-hour run finishes before the 3-hour point when the large job needs them.✓
- dYes, but only with a manual administrator approval each time.
Explanation:Backfill's whole point is running smaller jobs out of order when it provably won't delay the job ahead of them; a 2-hour job finishing before the deadline satisfies that.
Ti Cluster Scheduling ResourcesDifficulty 2
A job needs 8 GPUs total. Layout 1 places all 8 on a single node (connected by a fast intra-node link). Layout 2 places 2 GPUs on each of 4 separate nodes (connected by the cluster network, noticeably slower than the intra-node link). Assuming both layouts are otherwise identical, which one typically finishes the same synchronous training job faster, and why?
- aLayout 1, since synchronization stays on the fast intra-node link rather than the network.✓
- bLayout 2, since splitting GPUs across nodes multiplies available bandwidth.
- cBoth finish equally fast; gang scheduling only decides when ranks start, not step speed.
- dLayout 2, since more nodes give the job more CPU cores.
Explanation:The bottleneck is inter-GPU communication bandwidth for synchronization; Layout 1 keeps it on the fast link, and neither more nodes nor start time changes that.