Sample questions
Agents Tool UseDifficulty 3
During testing, an agent repeatedly calls the same tool with the same arguments across several iterations, and the returned observation is also unchanged each time, yet the agent has not produced a final answer. What does this most likely indicate, and what is a reasonable mitigation?
- aThis is expected behavior for any agent loop working as designed, and no change is needed, since repeated calls with an unchanged observation will resolve themselves given enough time.
- bThe tool itself must be non-deterministic, since a well-built deterministic tool would never be called more than once by a correctly functioning agent loop.
- cThe agent is likely stuck in a loop, misreading the observation or missing a stopping condition; detecting repeated identical calls and breaking out is a reasonable fix, alongside the step limit.✓
- dThis indicates the model's context window has been exceeded, and switching to a completely different tool is the only way to make the agent stop repeating itself.
Explanation:Identical repeated calls with an unchanged observation and no progress toward an answer is a classic sign the agent is stuck (e.g., misreading the observation or missing a stop condition), and a targeted repetition check is a useful mitigation in addition to a hard step limit. Calling this expected behavior, blaming tool non-determinism, or attributing it to context window size does not match the symptom described.
Agents Tool UseDifficulty 3
An agent has access to a delete_customer_account tool. Which design is safest for handling this specific tool within the agent loop?
- aTreat this exactly like a read-only tool such as
get_customer_info, since from the model's perspective both are simply tool calls it is free to request at any time. - bRequire explicit human confirmation (or a strong business-rule check) before executing this call, since it is destructive and hard to reverse, unlike read tools.✓
- cGive the tool a longer, more detailed description so the model is less likely to call it by mistake, and skip any runtime safeguard.
- dDisable step limits specifically for this tool so the agent has unlimited attempts to get it right.
Explanation:A destructive, hard-to-reverse action like account deletion warrants an extra runtime safeguard (confirmation or a business-rule gate) that a read-only tool does not need, because the consequence of a mistaken call is far more severe. Relying only on a better description, treating it identically to a read tool, or removing step limits does not add the protection this specific tool needs.
Agents Tool UseDifficulty 3
Why do teams commonly distinguish between 'read' tools (e.g., get_order_status) and 'write' or side-effecting tools (e.g., cancel_order) when designing an agent's tool set?
- aWrite tools can cause real, potentially irreversible changes if called incorrectly, so they often need stricter safeguards (confirmation, validation, idempotency) than read tools, whose worst case is usually just wrong or stale data.✓
- bRead tools always execute faster than write tools, because reading data is inherently a lighter database operation than writing it in every implementation.
- cThe model is technically unable to distinguish read operations from write operations within its schema format, so the application must track that distinction separately.
- dWrite tools cannot be described using the same schema format as read tools, since side-effecting operations require a fundamentally different parameter structure.
Explanation:The risk profile differs: a mistaken read only returns wrong information, while a mistaken write can cause a real, sometimes irreversible change, which is why write tools typically get extra safeguards like confirmation and idempotency. This distinction is about risk and consequence, not execution speed or schema-format limitations.
Agents Tool UseDifficulty 3
In one turn, a model requests two tool calls: get_weather(city="Paris") and get_weather(city="Tokyo"), where neither call's arguments depend on the other's result. In a separate scenario, a model requests get_user_id(email=...), and only in a later turn requests get_orders(user_id=...) using the ID returned by the first call. What is the key difference between these two situations?
- aThere is no real difference; every tool call must always wait for the previous call to finish, since the application processes tool calls strictly one at a time no matter what.
- bThe two
get_weather calls must run strictly one after another and can never be issued concurrently, because a single conversation turn only ever contains one independent action. - cThe
get_user_id and get_orders calls could safely be sent in the same turn, since batching dependent calls together is always more efficient regardless of missing input. - dThe two
get_weather calls are independent and could run in parallel, while get_orders depends on get_user_id's result, so it must wait for that observation first.✓
Explanation:Independent calls (the two weather lookups) have no data dependency between them and can potentially run concurrently, while a dependent call (get_orders needing the user ID) cannot be issued correctly until the earlier observation is available. Treating both cases the same, or batching a dependent call before its input exists, ignores this data-flow distinction.
Agents Tool UseDifficulty 3
A create_shipping_label tool internally calls a third-party carrier API that occasionally returns a different tracking number for what should be the same logical request (e.g., due to an internal retry on the carrier's side), even when called with an identical idempotency key. Why is this a problem for the agent's retry logic?
- aIt is not a problem, since sending an idempotency key always guarantees an identical output, no matter how the underlying carrier service is actually implemented internally.
- bIt only matters if the agent's step limit is set too low, and simply raising that limit will fully resolve the inconsistency in the tracking numbers being returned.
- cIt is irrelevant to the agent loop, because tool determinism only affects how many reasoning tokens the model consumes, not any real side effects that occur.
- dIf the tool can still produce a different real result on retry despite an identical key, the agent's assumption of safety is false, and retrying may cause duplicate side effects like an extra shipment.✓
Explanation:An idempotency key only helps if the underlying service actually honors it consistently; if the carrier can still produce a different real-world outcome on retry despite an identical key, the agent's assumption of safety is false and duplicate shipments can occur. Raising the step limit or claiming determinism is irrelevant to reasoning tokens does not address this genuine service-level gap.
Embeddings Vector SearchDifficulty 3
A junior engineer writes the following similarity function:
score = dot(query_vec, doc_vec)
The vectors are not normalized before this calculation. What problem can this cause?
- aThe dot product will always raise a runtime error whenever the two input vectors happen to have different dimensions.
- bVectors with a larger magnitude can score higher even when they are not more semantically similar, skewing the ranking.✓
- cThe result will always equal the cosine similarity value regardless of vector length.
- dThe score becomes negative whenever the two texts are unrelated in meaning.
Explanation:A raw dot product is sensitive to vector length, so documents whose embeddings happen to have larger magnitude can outscore more relevant documents with smaller magnitude vectors. Option (c) describes the opposite: an unnormalized dot product equals cosine similarity only when both vectors already have unit length.