Sample questions
Rag Reranking FusionDifficulty 1
Reciprocal Rank Fusion (RRF) combines two ranked lists by scoring each document as score(d) = sum over lists of 1/(k + rank_i(d)), where rank_i(d) is the document's 1-indexed position in list i. What role does the constant k play?
- aIt sets the maximum number of documents that can be fused from each list.
- bIt dampens top-rank influence, keeping scores from exploding at very small ranks.✓
- cIt converts BM25 scores into cosine similarity scores before fusion.
- dIt defines how many result lists must agree before a document is kept.
Explanation:k is added to the rank before inverting it. Without k, rank 1 would produce 1/1 = 1.0, an extremely large score compared to rank 10 (1/10 = 0.1) — a 10x gap. Adding k (e.g. 60) makes it 1/61 vs 1/70, a much smaller relative gap. So k smooths out the score curve across ranks, preventing the very top of one list from completely dominating the fused score.
Rag Reranking FusionDifficulty 2
Dense retrieval ranks document D1 at position 1. Keyword (BM25) retrieval ranks the same D1 at position 2. Using RRF with k=60, what is D1's fused score (score(d) = 1/(k+rank_dense) + 1/(k+rank_bm25))?
- aAbout 0.0325 (1/61 + 1/62).✓
- bAbout 0.0164 (only the dense list's contribution, 1/61).
- cAbout 1.5 (average of the two raw rank positions).
- dAbout 0.5 (rank 1 and rank 2 averaged and normalized to [0,1]).
Explanation:score(D1) = 1/(60+1) + 1/(60+2) = 1/61 + 1/62 = 0.016393... + 0.016129... = 0.032522..., i.e. about 0.0325. RRF sums a term per list where the document appears; it never averages raw positions or produces a value near 1.5 or 0.5.
Rag Reranking FusionDifficulty 3
Dense ranking: D1,D2,D3,D4,D5 (positions 1-5). BM25 ranking: D3,D1,D5,D2,D4 (positions 1-5). Using RRF with k=60, the fused scores come out to D1=0.032522, D3=0.032266, D2=0.031754, D5=0.031258, D4=0.031010. What is the correct fused ranking (highest score first)?
- aD3, D1, D2, D4, D5 — because D3 is rank 1 in BM25, ties always go to the keyword-search winner.
- bD1, D2, D3, D4, D5 — fusion just keeps the dense ranking unchanged since dense was listed first.
- cD1, D3, D2, D5, D4 — D1 edges out D3 since D1's ranks (1,2) sum higher than D3's (3,1).✓
- dD3, D5, D1, D2, D4 — BM25's ranking is used directly since it produced the top score for D3.
Explanation:Sorting the given scores descending: 0.032522 (D1) > 0.032266 (D3) > 0.031754 (D2) > 0.031258 (D5) > 0.031010 (D4). So the fused order is D1, D3, D2, D5, D4. Even though D3 is BM25's #1, D1 being #1 in dense and #2 in BM25 (a strong position in both lists) narrowly outscores D3 being #1 in BM25 but only #3 in dense.
Rag Reranking FusionDifficulty 3
A team fuses BM25 scores (raw range roughly 0-20, unbounded) directly with cosine similarity scores (range -1 to 1) by simple addition, and results look dominated by whichever retriever happens to produce larger raw numbers. Why does RRF avoid this failure mode?
- aRRF first converts BM25 scores to cosine similarity using a learned linear mapping, then adds them.
- bRRF discards raw scores and uses only each document's rank in each list, so no retriever's scale can dominate.✓
- cRRF clips every score to the [0,1] range using min-max normalization before summing.
- dRRF only works when both retrievers happen to already share the same scale, so the scenario described would make RRF fail too.
Explanation:RRF's formula, 1/(k+rank), takes only the rank (an integer position: 1st, 2nd, 3rd...), never the underlying score value. Since every retrieval method inherently produces a rank ordering regardless of its scoring scale, RRF sidesteps the scale-mismatch problem entirely — it never touches the raw BM25 or cosine numbers, so their different ranges can't skew the fusion.
Rag Reranking FusionDifficulty 2
Using the same two rankings, D1's fused score relative to D4's is computed at two different k values. At k=1: D1=0.8333, D4=0.3667 (ratio ~2.27x). At k=60: D1=0.0325, D4=0.0310 (ratio ~1.05x). What does this show about increasing k?
- aIncreasing k makes RRF ignore BM25 entirely and rely only on dense retrieval.
- bIncreasing k always changes which document ends up ranked first.
- cIncreasing k converts the fused scores into true probabilities that sum to 1.
- dIncreasing k flattens score differences across ranks, making fusion less sensitive to rank gaps.✓
Explanation:At k=1, the gap between rank 1 and rank 5 is proportionally huge (dividing by 2 vs 6). At k=60, adding 60 to every rank shrinks that same gap to dividing by 61 vs 65 — a much smaller relative difference. So larger k compresses (flattens) the score spread across ranks; it doesn't touch which retriever is used, doesn't produce a probability distribution, and doesn't guarantee a ranking change.
Rag Reranking FusionDifficulty 2
RRF is extended from 2 lists to 3 ranked lists (e.g. dense, BM25, and a recency-boosted list) by summing 1/(k+rank) across all three. With k=60 and ranks: A=(1,2,2), B=(2,1,3), C=(3,4,1), D=(4,3,4), the computed sums are A=0.048652, B=0.048395, C=0.047891, D=0.047123. What does this confirm about RRF?
- aRRF generalizes to any number of lists — it's a sum of one reciprocal-rank term per list.✓
- bRRF can only combine exactly two ranked lists at a time; three-way fusion requires running RRF twice in sequence.
- cWith three lists, k must be divided by 3 to keep the scores in the same numeric range as two-list RRF.
- dAdding a third list always increases every document's score proportionally, so the ranking order never changes.
Explanation:score(d) = sum_i 1/(k+rank_i(d)) is defined as a sum over however many ranked lists are supplied — nothing in the formula caps it at two. Here A wins with 0.048652 because it holds strong positions (1,2,2) across all three lists; the formula and the resulting order confirm RRF scales naturally to N lists without modification.