Sample questions
Dl Losses Output LayersDifficulty 1
A classifier's final layer produces a vector of raw scores, and softmax is applied along the class dimension. What is guaranteed about the resulting vector?
- aEach entry falls in (-1, 1) and the entries sum to zero.
- bThe largest raw score becomes exactly 1 and all other entries become exactly 0.
- cEach entry falls strictly between 0 and 1 and the entries along the class dimension add up to 1, so the vector can be read as a categorical distribution over the classes.✓
- dEach entry equals the raw score divided by the sum of the raw scores.
Explanation:Softmax exponentiates every score, which makes each entry strictly positive, then divides by the sum of those exponentials, which forces the total to 1. Because the exponential is positive and finite, mathematically no entry reaches exactly 0 or exactly 1, although in float32 a score far enough below the maximum does round to 0.0 and leaves its neighbour at 1.0. Hard-assigning 1 to the maximum describes argmax and one-hot encoding, not softmax, and dividing raw scores by their sum is a different normalisation that breaks on negative scores.
Dl Losses Output LayersDifficulty 2
In PyTorch 2.8 you run torch.softmax(torch.tensor([2.0, 1.0, 0.0]), dim=0) on CPU. Which output is closest to what is printed?
- a[0.333, 0.333, 0.333], the uniform distribution over the three classes
- b[0.500, 0.333, 0.167], i.e. proportional to the raw scores plus one
- c[0.667, 0.333, 0.000], i.e. proportional to the raw scores themselves
- d[0.665, 0.245, 0.090]✓
Explanation:Softmax computes exp(2), exp(1) and exp(0), which are 7.389, 2.718 and 1.000, then divides each by their sum 11.107. The result is roughly 0.665, 0.245 and 0.090. Because neighbouring scores differ by exactly 1, the ratio between neighbouring probabilities is e, about 2.718, which is the quickest way to recognise the right vector.
Dl Losses Output LayersDifficulty 2
Two logit vectors are fed to the same softmax: [1.0, 2.0, 3.0] and [51.0, 52.0, 53.0]. How do the two probability vectors compare, and why does the answer matter for the implementation?
- aThe second vector is far more peaked, because larger raw scores put more of the probability mass onto the maximum entry and squeeze the others toward zero.
- bThe second one is flatter, because the exponentials saturate.
- cThey differ only in the last decimal, because floating point rounds large exponentials.
- dThey are identical: softmax is unchanged when the same constant is added to every score, and implementations subtract the row maximum for exactly this reason.✓
Explanation:Adding a constant multiplies every exponential by the same factor, and that factor cancels between numerator and denominator, so the probabilities are untouched. Only the differences between scores matter, never their absolute level. Library implementations subtract the row maximum before exponentiating, which keeps the largest exponent at zero and prevents the sum from overflowing.
Dl Losses Output LayersDifficulty 3
A demand model predicts daily order counts. Its head is a single linear unit whose raw output z is read as the logarithm of the expected rate, and training minimises the Poisson negative log-likelihood exp(z) − y·z. On a day with y = 4 orders the head currently outputs z = 0.5. What does the gradient of that example's loss with respect to z equal, and which way does it push the head?
- a2·(exp(z) − y)·exp(z), which is −7.75 here, because the squared error between the rate and the count is chained through the exponential link.
- bexp(z) − y, which is −2.35 here, so the head is pushed up until the rate exp(z) reaches the observed count.✓
- c1 − y/exp(z), which is −1.43 here, since the log link makes the update depend on the ratio of count to rate rather than on their difference.
- dz − log(y), which is −0.89 here, since the log link makes the objective a residual on the log scale.
Explanation:Differentiating exp(z) − y·z with respect to z gives exp(z) − y, the predicted rate minus the observed count, and that is stationary exactly when exp(z) = y. Verified on PyTorch 2.8: F.poisson_nll_loss with log_input=True at z = 0.5 and y = 4 reported -0.3513 and an autograd gradient of -2.3513, which is exp(0.5) - 4. The log-input form drops the constant log(y!) term, which is why the reported value can be negative without anything being wrong, and the link keeps the predicted rate positive without any clipping on the output.
Dl Losses Output LayersDifficulty 2
A photo tagging model must attach any subset of 12 tags to an image; a single photo can legitimately carry 'beach', 'sunset' and 'people' at once. What output layer and loss fit this task?
- aOne output unit per tag with an independent sigmoid on each, trained with binary cross-entropy per tag, so several tags can be confidently on at the same time without competing for a shared budget.✓
- bOne output unit per tag with softmax over the tags, trained with categorical cross-entropy.
- cA single output unit whose value is rounded to the index of the strongest tag.
- dOne output unit per tag with softmax, then a threshold of 0.5 applied to each probability.
Explanation:Softmax ties the 12 outputs together by forcing them to sum to 1, so raising the confidence of one tag necessarily lowers the others: that is the wrong structural assumption for a multi-label problem. One sigmoid unit per tag turns each tag into its own yes/no decision, and binary cross-entropy scores each of those decisions separately, which is the structure a subset-valued label actually has. Thresholding a softmax at 0.5 makes the situation worse, because with 12 competing classes almost nothing ever crosses 0.5.
Dl Losses Output LayersDifficulty 1
A network classifies an image into one of 10 mutually exclusive categories. The penultimate feature vector has 256 entries and the loss is nn.CrossEntropyLoss. What should the final nn.Linear layer be?
- a
nn.Linear(256, 10), emitting one raw logit per category.✓ - b
nn.Linear(256, 9), because one category is implied by the other nine and adding it would be redundant. - c
nn.Linear(256, 10) followed by nn.Softmax(dim=1), so that the criterion receives proper probabilities. - d
nn.Linear(256, 1), because cross-entropy needs a single score to compare against the integer label.
Explanation:Cross-entropy needs one score per class so it can build a distribution over the 10 categories and pick out the target index, which means the layer must be 256 to 10 (2570 parameters including the bias). The criterion consumes raw scores and performs its own normalisation, so an explicit softmax layer in front of it is not what the interface expects. Dropping one class to nine outputs leaves the criterion unable to score that category at all, so it can never be predicted.