Sample questions
Evaluation TestingDifficulty 1
You ask an LLM the same question twice with temperature > 0 and get two differently worded but both reasonable answers. Why does this make evaluating LLM output harder than evaluating traditional software?
- aBecause non-determinism only affects creative-writing tasks, so factual question-answering is essentially unaffected by it
- bBecause LLMs always produce the exact same output for a fixed prompt, so evaluation is actually simpler than for traditional software
- cBecause a single run's output can't reliably represent the model's typical quality across attempts✓
- dBecause fixing the random seed once makes output fully deterministic afterward across every model and provider
Explanation:Since output varies run to run, one sample isn't representative of typical quality (c) — evaluation needs multiple samples or judge-based scoring tolerant of paraphrase. Non-determinism affects factual QA too, not just creative tasks (a is wrong). LLMs at temperature > 0 don't produce identical output each call, so it isn't simpler for that reason (b is wrong). Seed handling isn't guaranteed to fully eliminate variance across every provider or model (d is wrong).
Evaluation TestingDifficulty 1
In the context of evaluating an LLM application, what is a 'golden set' (eval set)?
- aA fixed collection of representative inputs paired with expected outputs, used to measure quality consistently✓
- bA set of the highest-scoring production requests, automatically saved whenever user feedback is positive
- cA list of banned words and phrases that a content filter checks every response against before returning it
- dA random sample of live traffic pulled fresh right before every deployment so the test data stays current
Explanation:A golden set is a curated, fixed benchmark with known expected outputs, reused across changes so scores are comparable over time (a). Automatically saved positive-feedback requests (b) describe an implicit feedback log, not a curated eval set. A banned-word list (c) is a guardrail/filter artifact, not an evaluation benchmark. Pulling fresh random traffic each time (d) removes the fixed baseline needed for consistent before/after comparison.
Evaluation TestingDifficulty 1
For a task where a model extracts a single structured value (e.g., an order ID) from a document, what is a key limitation of scoring outputs with 'exact match'?
- aIt only works for outputs written in English and produces errors on any other language
- bIt cannot be computed without first sending the output to a second LLM for review
- cIt requires the golden set to contain at least a million examples before it produces a meaningful score
- dIt scores 0 for any output that differs in formatting (e.g., extra whitespace or leading zeros), even if the value is correct✓
Explanation:Exact match is brittle to superficial formatting differences even when the value is functionally correct, so it often needs normalization or looser matching (d). Exact match is a plain string comparison, so it's language-agnostic (a is wrong) and needs no second LLM call (b is wrong). It also works fine on small golden sets; there's no minimum-example requirement (c is wrong).
Evaluation TestingDifficulty 1
What is the main practical difference between 'offline evaluation' and 'production monitoring' for an LLM application?
- aOffline evaluation is always performed by a human, while production monitoring is always fully automated
- bOffline evaluation runs against a fixed set of test cases before shipping; production monitoring observes live traffic after release✓
- cOffline evaluation only checks latency and cost, while production monitoring only checks correctness
- dOffline evaluation becomes unnecessary once production monitoring is set up, since live traffic eventually covers every case offline testing would
Explanation:The core distinction is timing and data source: offline eval uses a fixed pre-release test set, production monitoring watches live traffic continuously after release (b). Neither is restricted to human-only or fully-automated by definition (a is wrong). Both can track correctness, latency, or cost depending on setup (c is wrong). Production monitoring doesn't guarantee coverage of the specific edge cases offline testing deliberately targets, so offline eval remains useful (d is wrong).
Evaluation TestingDifficulty 1
What does the term 'LLM-as-judge' refer to in evaluation of LLM applications?
- aUsing a separate LLM call, guided by a rubric or reference answer, to score the system's outputs✓
- bA legal requirement that a model vendor's own model must certify the safety of any application built on top of it
- cA technique where the model being evaluated is asked to guess its own accuracy percentage in a single pass
- dA metric that counts how many times a user had to rephrase their prompt before getting a satisfactory answer
Explanation:LLM-as-judge means using a (typically separate) LLM call guided by criteria to score or rank outputs (a). There is no such vendor-certification requirement (b is fabricated). Self-reported confidence (c) is a different, generally unreliable technique, not judging another output against criteria. Counting rephrase attempts (d) describes a distinct usage metric, unrelated to judging output quality.
Evaluation TestingDifficulty 1
After changing a prompt template used in production, why is it good practice to re-run a suite of known test cases before deploying the change?
- aBecause prompt changes are not allowed to be deployed without written sign-off from the model provider
- bBecause re-running the test cases automatically reverts the prompt to the previous version if any case fails
- cTo catch cases where previously correct behavior has regressed, even if the change targeted something else✓
- dTo reduce the number of tokens the new prompt consumes compared to the old one
Explanation:Regression testing exists to catch unintended breakage in previously-working behavior, independent of what the new change was meant to fix (c). There's no such provider sign-off requirement (a is fabricated, provider-agnostic violation). Running a suite only observes results; it doesn't itself revert anything without separate tooling (b is wrong). Token count is a separate cost concern, not the purpose of regression testing (d is wrong).