Sample questions
Inference ServingDifficulty 1
In LLM inference, what does 'time to first token' (TTFT) measure, as opposed to total response latency?
- aThe total time needed to generate every output token in the response.
- bThe time from request receipt until the first output token is returned to the client.✓
- cThe time the server spends validating the request payload before it reaches the model.
- dThe average delay between two consecutive output tokens in the response.
Explanation:TTFT is specifically the delay before the first token arrives, dominated by prompt processing; total latency also includes generating every remaining token. Option a describes total latency, not TTFT; d describes inter-token latency, a separate metric; c describes an unrelated pre-processing step.
Inference ServingDifficulty 2
A team notices that requests asking for a one-sentence answer return in under a second, but requests asking the same model to write a 2000-word report take much longer, even though the input prompt length is similar in both cases. What best explains this?
- aThe model re-tokenizes the entire input prompt on every single output step.
- bLonger responses trigger extra rate-limit checks that shorter responses skip.
- cThe server compresses long responses, and client-side decompression adds the delay.
- dGeneration latency scales mainly with the number of output tokens, generated one at a time.✓
Explanation:Token generation is autoregressive: one token per decode step. More output tokens means proportionally more sequential steps, so latency grows with output length even when input length is similar. a is wrong because the input prompt is processed once, not re-tokenized per output token; b and c invent mechanisms that don't reflect how serving actually works.
Inference ServingDifficulty 1
Why do many LLM applications stream tokens back to the user as they are generated, instead of waiting for the full response and sending it all at once?
- aIt reduces the perceived wait time by showing output as the first tokens arrive.✓
- bIt reduces the total number of tokens the model needs to generate for the same answer.
- cIt guarantees a lower total cost per request than a non-streamed response.
- dIt lets the model skip tokens it already generated in a prior, unrelated request.
Explanation:Streaming improves perceived latency because the user sees the first tokens (TTFT) quickly, even though total generation time and total token count are unchanged. b, c, and d make false claims about how streaming affects token count, cost, or reuse across requests.
Inference ServingDifficulty 2
A client consumes a streaming completion response using the pseudocode below:
full_text = ""
for chunk in stream_response():
full_text += chunk.token
render_to_user(chunk.token)
return full_text
What does this loop correctly rely on for a good user experience?
- aIt must collect every chunk into full_text before any rendering can occur.
- bstream_response() blocks until the whole response is generated, so each chunk holds the full answer.
- cEach incoming chunk is rendered to the user as soon as it arrives.✓
- drender_to_user recomputes full_text from scratch on every iteration.
Explanation:The loop calls render_to_user(chunk.token) inside the same iteration it receives each chunk, so output appears incrementally. a and b contradict the point of streaming (they'd defeat the low-latency benefit), and d describes unnecessary recomputation that isn't in the code.
Inference ServingDifficulty 2
A self-hosted inference server exposes this config:
max_batch_size: 16
batch_timeout_ms: 20
What does this configuration most likely control?
- aThe maximum output length before a single request's response is truncated.
- bHow many requests the server groups into one forward pass, and how long it waits to fill that group.✓
- cThe number of retries before the server returns a rate-limit error.
- dThe maximum payload size, in megabytes, the server accepts per request.
Explanation:Batching groups concurrent requests to share compute in a single forward pass; batch_timeout_ms bounds how long the server waits to accumulate requests before running the batch anyway. a, c, and d assign unrelated meanings to the same field names.
Inference ServingDifficulty 2
An inference server enables aggressive request batching to increase throughput. What is the most likely downside for an individual request arriving right after a batch has just started processing?
- aIt may need to wait for the current batch to finish before its own batch can start.✓
- bIt is silently dropped rather than added to any batch.
- cIts output quality is lower, since batching changes token probabilities for that request.
- dIt bypasses the queue and is processed ahead of the already-running batch.
Explanation:Batching trades some per-request latency (waiting to join or start the next batch) for higher aggregate throughput; it doesn't drop requests, doesn't meaningfully change output quality (the same weights and math are applied per request; batching can introduce tiny floating-point differences, but that is not a quality change), and doesn't grant priority.