The assistant’s answer is polished, specific, and wrong. It says the customer’s plan includes priority support because it saw a similar plan name in an old document. The current account record was never retrieved, but the model filled the gap with a plausible fact.
An assistant that says “I don’t have enough information” can be more useful than one that always produces an answer. The challenge is making that admission actionable rather than turning the product into a machine that refuses whenever a question is difficult.
To build an assistant that recognizes missing context, separate what it knows from what it inferred, define which sources are authoritative, identify the specific uncertainty, and give the user a next step. Do not rely on a confidence score or a prompt instruction alone. The application needs a context contract and tests for when the system should answer, clarify, retrieve, or hand off.
“I don’t know” is not one state
An assistant can lack context in several different ways:
- Missing user detail: the user has not supplied an account, date, or preference.
- Missing application data: a required record could not be retrieved.
- No evidence: available sources do not answer the question.
- Conflicting evidence: two sources disagree or have different freshness.
- Unsupported task: the request requires a capability the assistant does not have.
- Tool failure: the system could not check the source, so it does not know whether data exists.
- Uncertain interpretation: the user’s request or the source data is ambiguous.
Those states should not collapse into a generic “I’m not sure.” If the account ID is missing, ask for it. If the tool timed out, say the account could not be checked. If two policies disagree, surface the conflict or route it to an owner. If the request is out of scope, explain what the assistant can do instead.
The difference matters because “no information found” and “could not retrieve information” are not the same claim. One is about the source; the other is about the system’s ability to consult it.
Define the evidence contract for an answer
Before asking a model to answer a question, define what evidence is needed and which source wins when sources conflict.
For an order-status assistant, the contract might be:
- status must come from the order service;
- delivery estimate may be shown only if the carrier returns one;
- customer profile data cannot substitute for the order record;
- if the lookup fails, do not infer status from the last conversation;
- if the result is stale, say when it was last verified.
This contract can be partly enforced in code. The application can require a successful lookup result before invoking the answer-generation step. It can include source IDs and timestamps in context. It can reject a response that includes a status field when no status was retrieved.
The model still needs instructions about using evidence, but the system should not depend solely on those instructions. If a required source is absent, the application can route to clarification or a service error before generating a confident answer.
1. Ask a question only when the answer can change the outcome
Clarifying questions are useful when a missing field blocks a decision. They are frustrating when the assistant asks for information it does not actually need.
For example, “Which order do you mean?” is appropriate when the user has multiple recent orders and the chosen order changes the answer. Asking for the order ID when the user is already on that order’s page is unnecessary; the application can pass the selected record reference in trusted context.
Design clarification around the missing variable:
- identify which field or choice is ambiguous;
- explain why it matters in plain language;
- ask one focused question;
- preserve the user’s original request;
- resume the same task when the user responds.
Do not ask a broad “Can you provide more context?” if the system can name what is missing. “Which date range should I use?” is actionable. “Please clarify” shifts the reasoning burden back to the user.
2. Distinguish empty retrieval from failed retrieval
Retrieval-based assistants need explicit tool status. A search that succeeds with zero authorized results is different from a search service timeout, and both differ from a query rejected because the user lacks access.
Represent those outcomes explicitly:
retrieval_status = found | no_results | denied | unavailable | invalid_query
Then define behavior for each:
found: answer using the returned sources and identify freshness where needed;no_results: say no matching source was found and offer a search refinement;denied: state that the user lacks access or provide an access request path without revealing the record;unavailable: explain that the assistant could not check and offer a retry or manual route;invalid_query: ask a targeted clarification or normalize the query.
Never translate a timeout into “there is no information.” That answer trains users to believe a tool outage is a fact about the business.
3. Make sources visible and freshness legible
When an answer depends on documents or system records, show enough provenance for the user to judge it. A citation should link to the actual authorized source or identify the record and its last-updated time. A citation that points to a search result but not the relevant passage does not help much.
Freshness is especially important for mutable facts: prices, policy windows, account status, access rights, inventory, and schedules. Include the source timestamp in the context, and decide what staleness limit is acceptable. If the source is older than that limit, either refresh it or qualify the answer.
Do not use a citation as a confidence decoration. The assistant should not cite a document that merely mentions the same topic but does not support the specific claim. Evaluation should check claim-to-source support, not just whether a citation exists.
Define source precedence before sources disagree
An assistant that uses several systems needs a rule for conflicts. A CRM field may be updated by a user, while a billing service is authoritative for payment status. A policy document may be newer than a cached summary. The model should not decide which system “feels more trustworthy” from the prose.
Define source precedence in application code or in a versioned policy. For each fact type, record the authoritative source, acceptable freshness, and whether a conflicting source should trigger a review. Pass those facts and their provenance to the model only after the application has applied access control.
If two equally authoritative sources disagree, do not silently choose the newest timestamp unless the business rule says that is valid. Surface the conflict, ask a domain owner, or return a qualified answer that names the discrepancy. This avoids turning an unresolved data-quality issue into a confident model claim.
Distinguish stale context from unavailable context
A tool can return a successful response that is stale. An account snapshot from yesterday is available data, but it may not answer whether access is still active now. A cache hit is not proof of freshness.
Include retrieved_at, source version, or record version in the context. For sensitive or mutable facts, re-fetch immediately before answering or executing. If the source cannot meet the freshness requirement, classify the result as stale and do not present it as current.
This matters especially for write workflows. The assistant may prepare a change using one state, while another user updates the record before approval. Re-read the entity at the point of execution and compare versions. If the relevant state changed, invalidate the old proposal and ask for a new review.
Stale data should have its own user-facing state. “I found an older result” is different from “I could not find a result,” and both are different from a current, verified answer.
4. Treat confidence as a signal, not a truth value
Models can produce confidence-like language or numbers, but those values are not automatically calibrated probabilities. “I’m 95% sure” may sound precise without corresponding to a measured 95 percent success rate.
Prefer observable signals:
- required source retrieved successfully;
- all required fields present;
- sources agree on the relevant fact;
- schema validation passed;
- request falls within evaluated input types;
- tool result is recent enough;
- no policy or permission boundary was hit.
Use model confidence only if it is calibrated on representative data for the exact task and threshold. Even then, combine it with deterministic checks. A high score cannot authorize a payment or bypass a missing source.
5. Separate uncertainty in the answer from uncertainty in the action
The assistant may be unsure how to phrase a response while the underlying data is clear. Or the prose may sound confident while the action is uncertain. Track these separately.
For a write workflow, the model can prepare a proposal with fields and sources. Application code validates the target, business rule, and permission. If the proposal is incomplete, ask for missing details. If the action is high-impact, require approval even when the model’s natural-language explanation sounds certain.
The user-facing answer should not imply completion until the tool confirms it. “I can prepare the change” differs from “The change is complete.” If the tool result is unknown after a timeout, say that the system is checking status rather than making a new claim.
Ask only for information the user can supply
Clarification should not ask the user to repair an internal data or service failure. If the assistant cannot reach the order system, asking “What is your order status?” merely transfers the system’s job to the user. If an account identifier is missing, asking for the identifier may be reasonable. If an authorization check fails, requesting a password in chat is not.
Before asking, check whether the application already has the value in a trusted context. Before asking again, check whether the answer is already in the conversation. Repeatedly requesting a detail the user supplied is a continuity bug, not prudent uncertainty handling.
If the user does not answer, preserve the pending task only as long as necessary. Expire it according to the product’s retention rules, and do not keep background workers polling. When the user returns, show what is known and ask only for the still-missing detail.
Clarifying questions should be accessible and easy to answer. Offer a short selection when the ambiguity is among a few known options; use free text when the user’s situation is genuinely open-ended. Do not make the model ask three separate questions when one well-structured prompt can collect the required fields.
Calibrate abstention against the cost of guessing
Abstention is a decision with a cost. Asking a user one extra question may be cheap; sending every invoice to manual review may make the feature unusable. Conversely, an unsupported answer about an account restriction may have a high consequence. Set thresholds by task and impact rather than using one global “confidence” rule.
Review examples near the decision boundary: cases the assistant answered despite missing evidence, and cases it escalated even though the answer was present. Label why each decision was correct or incorrect. If the model abstains because retrieval is empty, fix retrieval coverage. If it abstains because the user’s wording is ambiguous, improve clarification. If it abstains on a prohibited action, keep the policy boundary.
Track the resolution of abstained cases. Did the user provide the missing information? Did a human find the answer? Was the request abandoned? Those outcomes show whether the assistant is appropriately cautious or simply moving work into another queue.
6. Build an output schema with an explicit uncertainty path
For some tasks, structured output helps the application distinguish an answer from a request for clarification or a lack of evidence.
from enum import StrEnum
from pydantic import BaseModel
class Outcome(StrEnum):
ANSWER = "answer"
CLARIFY = "clarify"
INSUFFICIENT_EVIDENCE = "insufficient_evidence"
SOURCE_UNAVAILABLE = "source_unavailable"
OUT_OF_SCOPE = "out_of_scope"
class AssistantResult(BaseModel):
outcome: Outcome
answer: str | None = None
question: str | None = None
source_ids: list[str] = []
reason_code: str | None = None
The application should validate that the fields match the outcome. For example, CLARIFY should contain a question; ANSWER should have the required source IDs for a grounded workflow; SOURCE_UNAVAILABLE should not masquerade as INSUFFICIENT_EVIDENCE.
The schema does not make the model honest. It makes the application’s handling of different outcomes explicit and testable. Check the source IDs against the actual retrieval results before rendering the answer.
7. Evaluate the cases where “admit uncertainty” is correct
An eval set should include answerable, unanswerable, ambiguous, and unavailable cases. If every test expects a direct answer, prompt tuning will teach the assistant to guess. If every difficult case expects refusal, the assistant will become useless.
For each example, label:
- whether evidence exists;
- whether the source was available;
- whether clarification would resolve the uncertainty;
- whether the assistant should answer, clarify, decline, or escalate;
- which claims must be supported;
- what a useful next step is.
Measure both sides:
- Unsupported-answer rate: the system answered without adequate evidence.
- Unnecessary abstention rate: the system declined despite sufficient evidence.
- Clarification efficiency: whether the question asks for the specific missing information.
- Source failure accuracy: whether tool outages are reported as outages, not as absent facts.
- Task completion: whether the user can resolve the request after the clarification or handoff.
OpenAI’s evaluation guidance recommends task-specific datasets, human calibration, and continuous evaluation because model outputs vary. The dataset should include real input variation and should grow from production failures, with sensitive data handled under an explicit retention policy. (OpenAI, evaluation best practices)
8. Avoid teaching the model to hedge every sentence
An assistant that adds “I may be wrong” to every response is not calibrated. It is noisy. Users stop noticing the caveat, and the system still may not identify the cases where uncertainty matters.
Use uncertainty language tied to evidence:
- “I found two matching accounts; which one do you mean?”
- “The policy document I found was last updated in March. I can’t verify whether it is current.”
- “I couldn’t reach the order system, so I can’t confirm the delivery status.”
- “The sources disagree about the renewal date; I’ve routed this for review.”
These statements say what is missing and what happens next. Avoid unsupported numerical confidence and vague disclaimers that do not help the user decide.
9. Preserve uncertainty in summaries and memory
When an assistant stores a conversation summary or durable memory, do not turn an uncertain statement into a fact. Keep provenance and status:
fact: renewal date = 2026-11-01
source: CRM record 8431
verified_at: 2026-10-22
status: verified
For an unresolved claim, store it as a question or disputed value, not as canonical state. If a later session reads “customer said renewal may be November,” it should not treat that as a confirmed contract date.
Summaries are lossy. Keep the source reference for facts that affect decisions, and refresh the value from the authoritative system before a high-impact answer. A model-generated memory should never outlive the policy or record it summarizes without a freshness check.
10. Design the recovery path alongside the uncertainty state
“I don’t know” should lead somewhere. Depending on the case, the next step may be:
- ask the user for a missing identifier;
- let them select among candidate records;
- retry a transient tool failure with a bounded policy;
- show the relevant source document;
- create a human review task;
- provide a manual workflow or contact route;
- stop because the request is prohibited.
Make the next action explicit and safe. Do not let “try again” generate a new payment, resend an email, or run a broader query. For actions that may have completed, reconcile the previous operation before offering a retry.
The application should also handle the case where clarification never arrives. Expire the pending task or let the user resume it later; do not keep workers polling or retain sensitive context indefinitely. Tell the user whether their draft was saved and how long it remains available.
Log why the assistant withheld an answer
Record a stable reason code such as missing_identifier, no_authorized_source, source_conflict, or retrieval_unavailable, along with the source references and prompt version. Do not log the full private conversation by default. These reason codes let product and support teams distinguish a cautious assistant from a broken data path and decide which issue deserves attention.
A release checklist for honest uncertainty
Before launch, verify:
- The assistant distinguishes no results from retrieval failure and access denial.
- It asks targeted questions when a specific missing field blocks progress.
- It cites or identifies sources for consequential claims.
- It checks source freshness for mutable facts.
- It does not turn a model confidence score into authorization.
- Structured outcomes are validated against their required fields.
- Evaluation covers answerable, ambiguous, unavailable, and unanswerable cases.
- The UI explains the next step and preserves user work.
- Unknown side-effect outcomes are reconciled before retry.
- Summaries and memory preserve provenance and uncertainty status.
The right kind of honesty is specific. “I don’t know” is useful when the assistant can explain what it lacks, why that matters, and how the user can proceed. Build that behavior into the data path, tool results, output contract, and recovery workflow. Then the assistant can be direct when evidence is strong and appropriately cautious when it is not—without turning every answer into a hedge or every gap into a dead end.