Sample questions
Dl Initialization OptimizersDifficulty 1
A fully connected hidden layer is initialised so that every unit in the layer receives exactly the same weight vector and the same bias. Training then proceeds with plain gradient descent. What happens to those hidden units as training continues?
- aEach unit slowly drifts towards a different weight vector, because the mini-batches seen during training differ from one step to the next.
- bThey keep receiving identical gradients, so the layer behaves like a single unit copied many times.✓
- cOnly the first unit keeps learning; the rest stay frozen at their initial values.
- dThe units diverge as soon as the non-linearity is applied, because ReLU responds differently to each unit's pre-activation.
Explanation:Units that share the same incoming weights compute the same pre-activation for every input, so the loss is an identical function of each of them and back-propagation hands them identical partial derivatives. Identical parameters plus identical gradients means identical updates forever, so the layer's whole output space collapses onto one direction and the extra width buys nothing at all.
Dl Initialization OptimizersDifficulty 2
In PyTorch 2.8 on CPU, a two-layer MLP Linear(3,4) -> ReLU -> Linear(4,1) has every weight matrix and every bias set to exactly zero. One batch is pushed through, an MSE loss is computed and backward() is called. Which parameters end up with a non-zero gradient?
- aBoth weight matrices, because the input batch itself is non-zero and reaches the first layer unchanged.
- bThe second layer's weight and bias, since the loss is computed directly from that layer's output.
- cEvery bias in the network, while the two weight matrices are the only tensors stuck at a zero gradient.
- dOnly the output layer's bias, which is the single parameter still exposed to the error.✓
Explanation:The hidden activations are relu(0) = 0, and the gradient of the second weight matrix is built from those activations, so it vanishes. The gradient that would reach the first layer has to travel back through the second weight matrix, which is also zero, so it vanishes as well. The output bias is the only parameter whose gradient is the raw error itself, which is why an all-zero network never leaves that fixed point.
Dl Initialization OptimizersDifficulty 1
Xavier (Glorot) initialisation draws the weights of a layer from a distribution whose variance is tied to the layer's shape. Which quantity does that variance target?
- aRoughly 2 divided by the sum of fan_in and fan_out, so that the forward and backward requirements are traded off against each other.✓
- bRoughly 2 divided by fan_in alone, matching the number of incoming connections that are summed into each unit's pre-activation value.
- cRoughly the reciprocal of the batch size, so the variance shrinks as batches get bigger.
- dRoughly 1 divided by the product of fan_in and fan_out, which keeps the total weight energy fixed.
Explanation:The Glorot derivation asks for the variance of the activations to be preserved going forward, which wants 1/fan_in, and for the variance of the gradients to be preserved going backward, which wants 1/fan_out. A layer cannot satisfy both unless it is square, so the rule takes the harmonic compromise 2/(fan_in + fan_out). Batch size never enters the derivation because it is a property of the data pipeline, not of the layer.
Dl Initialization OptimizersDifficulty 2
He (Kaiming) initialisation uses a variance of about 2/fan_in for a layer followed by ReLU, whereas the same derivation without an activation would ask for 1/fan_in. Where does the extra factor of 2 come from?
- aFrom the fact that ReLU has a slope of 2 on the positive side, which doubles the signal that passes through it.
- bFrom the two weight matrices involved in one back-propagation step, one for the forward pass and one for the backward pass.
- cReLU zeroes out roughly half of a symmetric pre-activation distribution, halving the mean square of the layer output, so the weights are scaled up to compensate.✓
- dIt compensates for the bias vector, which contributes a second independent source of variance to the layer output.
Explanation:For a zero-mean pre-activation z, the mean square of relu(z) is exactly half the mean square of z, because the negative half of the distribution is mapped to zero and the positive half is passed through untouched. Without correction the signal's second moment would halve at every layer and decay geometrically with depth. Doubling the weight variance restores it, which is exactly what the 2/fan_in rule does.
Dl Initialization OptimizersDifficulty 2
A 12-layer ReLU network with 256 units per layer is initialised with Xavier instead of He, and the mean square of the activations is measured layer by layer on the very first forward pass. What pattern does that measurement show?
- aThe mean square shrinks roughly by half at every layer, so by the deepest layer it is a tiny fraction of the input value.✓
- bThe mean square grows steadily with depth, roughly doubling each layer, because Xavier is too aggressive for rectified units.
- cThe mean square is unchanged from layer to layer, because Xavier and He differ only in the backward pass.
- dThe mean square stays flat for the first few layers and then grows without bound once ReLU starts saturating in the deeper part of the stack.
Explanation:For a square layer Xavier gives a weight variance of 1/fan_in, which is exactly half of what a rectified layer needs, and ReLU then throws away half of the second moment on top of that. The two effects compose into a factor of about one half per layer, which is a geometric decay in depth. A measured run over twelve layers ends several orders of magnitude below the input, which is why deep rectified stacks want the larger He scale.
Dl Initialization OptimizersDifficulty 1
In plain stochastic gradient descent the update applied to a parameter is minus the learning rate times its gradient. If the learning rate is doubled while everything else is held fixed, what happens to that single update?
- aIts length doubles and its sign flips, since the update is a subtraction.
- bIts length is unchanged but it rotates towards the steepest coordinate of the gradient.
- cIts length quadruples, because the loss is a quadratic function of the step size.
- dIts length doubles and its direction is unchanged.✓
Explanation:Plain SGD scales the negative gradient by a single scalar, so the learning rate controls only how far the step goes, never where it points. Doubling a positive scalar doubles the magnitude and leaves the direction alone. The quadratic behaviour of the loss shows up in how much the loss changes, not in how large the parameter update is.