Sample questions
Dl Sequence AttentionDifficulty 1
In a vanilla recurrent network unrolled over a sequence of length T, how are the recurrent weight matrices used across the T steps?
- aA distinct recurrent matrix is learned per step, so a longer sequence needs proportionally more parameters
- bOnly the input projection is shared; a fresh recurrent matrix is allocated for every step of the sequence
- cThe same matrices are reused at every step, so the parameter count is independent of sequence length✓
- dThe matrices are shared across the batch but resampled from the initializer at each step of the sequence
Explanation:Recurrence means one cell function is applied repeatedly, so the same input-to-hidden and hidden-to-hidden matrices act at every step. That is exactly why the same trained network can process sequences of any length, and why what makes a recurrent layer larger is a wider hidden state rather than a longer input.
Dl Sequence AttentionDifficulty 2
PyTorch 2.8. lstm = nn.LSTM(input_size=6, hidden_size=10, batch_first=True) and x has shape (3, 7, 6). After output, (h_n, c_n) = lstm(x), what are the shapes of output and h_n?
- a
output is (3, 7, 10) and h_n is (1, 3, 10)✓ - b
output is (3, 7, 6) and h_n is (3, 10) - c
output is (7, 3, 10) and h_n is (3, 1, 10) - d
output is (3, 10) and h_n is (3, 7, 10)
Explanation:The layer emits one hidden vector per timestep, so with batch_first=True the stacked outputs keep the batch and time axes and swap the feature axis to hidden_size. The final state is returned separately and is indexed by layer-direction first, which is why it carries a leading axis of size 1 for a single unidirectional layer.
Dl Sequence AttentionDifficulty 3
You build nn.LSTM(input_size=8, hidden_size=16) in PyTorch 2.8 and sum p.numel() over its parameters. Which count do you get, and why?
- a416, because one weight block of shape (16, 8) plus one of shape (16, 16) plus a bias covers the cell
- b832, because two gates need a full input and recurrent projection each plus their biases
- c1664, because four gates each need an input projection, a recurrent projection and two bias vectors✓
- d1600, because four gates need input and recurrent projections but the cell keeps a single bias vector
Explanation:The layer packs the four gate pre-activations into one block, giving weights of shape (416, 8) and (416, 16), and PyTorch keeps two separate bias vectors of length 416 for the input and recurrent sides. That is 4(168 + 1616 + 2*16) = 1664, measured directly on the module.
Dl Sequence AttentionDifficulty 2
For the same input and hidden sizes, how does a GRU's parameter count compare with an LSTM's, and what structural fact explains the ratio?
- aThey are equal, because a GRU uses the same four gates but ties the input and forget gates together
- bA GRU has three quarters as many, because it has three gated blocks instead of four✓
- cA GRU has half as many, because it drops both the cell state and the output gate
- dA GRU has more, because merging memory into the hidden state needs an extra projection
Explanation:An LSTM computes four gated blocks of pre-activations while a GRU computes three, and each block costs the same input projection, recurrent projection and bias. Measured with input 10 and hidden 20, the GRU came out at 1920 against the LSTM's 2560, exactly a three-to-four ratio.
Dl Sequence AttentionDifficulty 2
A recurrent stack is built with two layers running over the same sequence. At step t, what does the second layer take as its input?
- aThe first layer's final hidden state, broadcast unchanged to every step of the second layer
- bThe first layer's hidden output at step t; depth and time remain separate axes✓
- cThe raw input at step t again, with the two layers meeting only when their final states are concatenated
- dThe concatenation of every first-layer output from step 0 up to step t, which is why the width grows with t
Explanation:A stack is a recurrence in depth applied to the per-step output sequence of the layer below, so the second layer consumes one vector per timestep exactly as the first consumes one input per timestep. Rebuilding a two-layer module as two single-layer ones and feeding the first one's output sequence into the second reproduced the stacked module's output to float tolerance.
Dl Sequence AttentionDifficulty 3
PyTorch 2.8. A self-attention layer has separate learned query and key projections. You softmax the scaled scores for a 10-token input and look at the weight matrix. How often should the largest weight in a row sit on the diagonal, that is on the token's own position?
- aAlways, because a vector's dot product with itself is the largest entry its row can hold
- bUsually, because a token's query and its key are two images of one vector and stay correlated
- cNever, because a token is excluded from its own key set and cannot read itself
- dNo more often than any other column, since query and key are different images of the token✓
Explanation:The query and the key for one token are two different linear images of it, so the diagonal score is an ordinary dot product between two unrelated vectors and carries no built-in advantage. Across five random draws with 10 tokens and width 16 the diagonal held the row maximum in 0 to 2 rows out of 10 and the mean diagonal weight stayed around the uniform value of 0.1; feeding the raw token vectors in as both query and key instead put the maximum on the diagonal in all 10 rows, which is where the intuition comes from.