Designing AI features for users who do not trust AI

An accounts payable team gets an AI feature that extracts invoice totals and writes them into the ERP. The pilot is a success by every product metric: adoption is high, the model is accurate on the sample, and the team says the tool is useful. Six weeks later, finance discovers that two people have been re-keying every extracted total into a spreadsheet before approving, because they don't believe the numbers. The feature saved twenty minutes and created thirty minutes of verification.

That's not an adoption problem, and it isn't fixed by a better onboarding tour. It's a design problem: the feature asked for trust it hadn't earned, and the users responded rationally by building their own control system around it.

Distrust is often treated as an obstacle to overcome with clearer copy and friendlier illustration. That framing is backwards. For consequential work, distrust is a sensible prior. The design job is not to make people trust the model. It is to make the feature safe enough to use without trust, and then to give users evidence that lets their trust calibrate to reality.

Distrust is a rational response to asymmetric consequences

Consider what a user is being asked to accept. A system they cannot inspect produces an answer they cannot verify cheaply, on a task where a mistake costs them personally — a re-opened month-end, an unhappy client, a compliance finding. The vendor's incentives favor shipping. The model's failure is silent by nature: it produces fluent, plausible output whether or not it's right. And the user has probably already been burned by a different AI feature that was demoed as "95% accurate" and turned out to be 95% accurate on a distribution that didn't include their work.

Under those conditions, skepticism is not technophobia. It's an accurate read of the risk. A product that treats it as ignorance will keep losing to a spreadsheet, because the spreadsheet is inspectable, deterministic, and owned by the user.

There's a second failure mode that gets less attention: too much trust. Research on automation bias finds that people accept automated suggestions even when they're wrong, especially under time pressure and high workload (automation bias review). A feature designed to maximize trust can produce a team that stops checking — which is worse than a team that checks too much, because the errors become invisible.

So the target is neither trust nor distrust. It's calibrated reliance: the user relies on the system where it's reliable, verifies where it isn't, and can tell the difference.

Start by naming what the system can't do

Most AI product copy describes capability. Very little describes limits. That asymmetry is the first thing to fix.

The Microsoft Guidelines for Human-AI Interaction put this first: make clear what the system can do, and make clear how well it can do it (HAX Toolkit). The second guideline is the one teams skip. "Extracts invoice totals" is a capability statement. "Extracts totals from text-based PDFs; may misread scanned invoices and multi-currency totals; does not verify against the purchase order" is a limits statement, and it's what a skeptical user needs.

Practical ways to make limits concrete:

Name the input types that work. "Works with digital PDFs. Scanned documents are flagged for review." Users can then predict their own failure cases.
Show the failure rate in the unit users care about. Not "97% accuracy" but "about 3 in 100 invoices need a correction, mostly scanned or foreign-currency ones." If you don't have that number for a segment, say you don't.
Refuse visibly. When the model is uncertain or the input is out of distribution, the UI should say "I can't read this one" rather than produce a low-quality guess. A system that admits limits is more credible when it claims success.
Avoid anthropomorphic framing. "The assistant thinks this is a duplicate" invites a mental model of a colleague. "Matched on invoice number and amount" invites a mental model of a check. The second is easier to trust appropriately.

💡 Practical note: the limits statement is also a product spec. If you can't write down three things the feature does badly, you probably haven't tested it on enough real inputs.

Make the first interaction cheap, reversible, and small

Trust is built by repeated low-stakes success, not by a claim. Design the first use so that being wrong costs almost nothing.

For invoice extraction, that means starting with a read-only view: here is what I found, here is the source region, nothing has been written. The user can compare against the PDF without changing any state. Only after they accept a few items does the feature earn the right to write.

This sequencing is sometimes dismissed as friction. It's actually risk transfer: the user gets evidence before exposure. The alternative — write first, undo later — asks them to trust the undo path, which they have no reason to believe exists until they've tested it.

A useful pattern is a graduated permission ladder:

Suggest only, no persistence.
Pre-fill a form the user submits.
Write to a staging table that requires approval.
Write directly, with an audit entry and a one-click revert.

Move users up the ladder based on their behavior, not on a timer. A user who accepts 50 extractions without edits has demonstrated a reason to reduce friction. A user who edits 30% of them needs a different UI, not a nudge.

Show the receipt, not the reasoning

Users who don't trust AI are usually not asking for a chain-of-thought transcript. They're asking: what did you look at, what did you ignore, and how would I check?

A receipt is more useful than an explanation. For an extraction, it's the page number and bounding box. For a retrieval answer, it's the source documents with the relevant passage highlighted. For a classification, it's the fields that drove the label and the ones that were missing.

This is where "explainability" often goes wrong. Teams expose a model-generated rationale, which is itself unverified text. A skeptical user reads it and correctly concludes that a plausible explanation is not evidence. The receipt is evidence because it points at something outside the model.

Concretely:

Link every claim to the artifact it came from, at the granularity the user can check (page, paragraph, row).
Show what was excluded: "I did not use the two attachments because they are images."
Show the version of the source. If the policy changed yesterday, the answer should say which version it used.
Show the time. A cached answer from last week is a different claim.

The Microsoft guidelines frame part of this as "make clear why the system did what it did" (G11). The practical version is narrower and stronger: make it possible to audit the output without reading the prompt.

Let users accept in pieces

All-or-nothing acceptance is a trust killer. If the feature extracted twenty line items and one is wrong, the user must either reject everything or accept an error. Most will reject everything, then stop using the feature, because the cost of correcting the batch exceeds the benefit.

Per-item acceptance changes the economics. The user accepts the nineteen correct items, fixes the one, and the system learns which items are hard. It also produces a much better signal: not "the user rejected the output" but "the user corrected line 14's quantity, and the source was a scanned table."

Design implications:

Make each item independently actionable, with a visible state (suggested, accepted, edited, rejected).
Default to not applying anything until accepted, at least for the first sessions.
Preserve edits as structured corrections, not free text, so they can be evaluated later.
Show a summary at the end: "19 accepted, 1 edited, 0 rejected." That summary is the user's own trust signal.

Make correction the fastest path

If correcting the AI is slower than doing the task by hand, the feature has no reason to exist. This sounds obvious and is routinely violated: correction lives in a modal, requires re-typing the whole field, or discards the user's edit on the next refresh.

The correction path should be at least as fast as manual entry, and ideally faster because the model got most of it right. That means inline editing, keyboard-first interaction, and no confirmation dialogs for changes the user just made deliberately.

Corrections should also stick. If the user fixes the same vendor's invoice format every month, the system should remember the pattern or at least stop making the same suggestion. "Learns from your corrections" is a claim that has to be true in a way the user can observe; otherwise it's worse than no learning, because it teaches users that their effort is wasted.

Fail in a way that preserves the user's work

The most trust-destroying moment in an AI feature is a failure that loses work. A timeout that clears the form. A retry that duplicates an entry. An error that discards a half-reviewed batch.

Design the failure states explicitly:

Keep the user's input and their in-progress decisions.
Show what succeeded and what didn't, per item.
Offer a retry that resumes rather than restarts.
Never silently substitute a different result. If the model was downgraded to a cheaper fallback, say so.
Put the error where the work is, not in a toast that disappears.

This is also where a "no AI" fallback matters. If the model is down, the user should be able to complete the task manually in the same screen, with the same data. A feature that becomes a hard blocker during an outage teaches users to route around it permanently.

Give users a global off switch that isn't punitive

Users who don't trust AI often want to use some of it. A single on/off toggle is too coarse; no toggle is worse. The useful controls are granular:

Turn off auto-apply but keep suggestions.
Turn off a specific category ("don't summarize customer emails; I'll read them").
Turn off for a specific client or project.
See and delete what the system remembers about their corrections.

The HAX guidelines call this "provide global controls" (G17) and "support efficient dismissal" (G8). The product translation is: opting out should be one click, should not require a support ticket, and should not degrade the rest of the product. If opting out feels like a punishment, users will assume the feature is mandatory and disengage from the whole surface.

Don't fake precision

Confidence scores are seductive and usually harmful. A number like "87% confident" implies a calibration the system rarely has, and users anchor on it. Worse, the number often comes from a model's self-report, which is not a probability.

If you show uncertainty, show it in a form the user can act on:

Categorical: "clear match," "possible match," "needs review."
Comparative: "this one is less certain than the previous five."
Evidence-based: "the total appears on page 2 and again on page 7 with a different value."
Action-oriented: "review this line" rather than "confidence: 0.62."

If you must show a number, calibrate it against outcomes and publish the calibration. Otherwise it's decoration that makes the UI look scientific while giving the user nothing to reason with.

Tell users when the system changes

A model upgrade, a prompt change, or a new retrieval index can shift behavior in ways the user experiences as unpredictability. If the feature silently gets better on average but worse on the user's edge case, they will conclude it's unreliable — correctly, from their point of view.

Notify on material changes, in plain language: "The extraction model was updated on 12 March. Scanned invoices should be more accurate; multi-currency handling is unchanged." Keep a changelog the user can reach. If behavior changes for a specific workflow, say so before the user discovers it.

The HAX guideline is "notify users about changes" (G18). The reason is not transparency theater. It's that unexplained behavior change is indistinguishable from unreliability.

Three kinds of skepticism, three different fixes

"Users don't trust AI" is too blunt to design against. In practice, skepticism comes from different places, and each one needs a different response.

The burned user. They've seen an AI feature fail in a way that cost them time or credibility. They are not opposed to automation; they are opposed to repeating a specific bad experience. Arguing about average accuracy will not help. What helps is segment-level honesty ("this is the case that broke last time, and here is what changed"), a visible failure path, and a trial on their own data where they control the accept step.

The accountability holder. They will sign off on the output, so they bear the consequence. Their objection is not epistemic but structural: "if this is wrong, I'm the one explaining it." They need an audit trail, a clear statement of what the system did versus what a human approved, and an approval step that records who approved what. Removing the human from the loop does not reassure them; it removes their defense.

The craft user. They can do the task well and suspect the feature will do it worse. Sometimes they're right. The productive response is not to insist on parity but to give them a genuine division of labor: let the model handle the tedious 80% and let them exercise judgment on the rest. If the feature is actually worse for their cases, let them turn it off for those cases without leaving the product.

Treating all three as one "trust problem" produces generic reassurance that persuades none of them. The burned user needs a controlled retry. The accountable user needs a record. The craft user needs a boundary they set.

Trust is also a property of the workflow

A trustworthy screen can sit inside an untrustworthy process. If the AI output enters an approval chain with no owner, if nobody can say who is responsible when it's wrong, or if the vendor can change the model without notice, users are right to distrust the feature regardless of how good the UI is.

The operational commitments that matter:

A named owner. Someone inside the company is accountable for the feature's behavior, not just its uptime.
A change policy. What triggers a model change, how it's evaluated, how users are told.
A retention policy. What inputs and outputs are stored, for how long, and who can read them. Users are often more worried about the transcript than the answer.
An incident path. What happens when the feature produces a harmful output. Not a support macro; a real process with a timeline.
A rollback. The ability to revert to the previous model or prompt without a deploy, and a stated trigger for doing so.

These are not abstract governance. They show up in user behavior. A team that knows the feature can be rolled back is more willing to report failures. A team that suspects reports disappear into a backlog stops reporting and starts working around the tool.

🔍 Why this matters: users infer the system's reliability from the organization's response to failure. A fast, specific response buys more trust than a year of clean metrics.

The onboarding mistake: teaching capability instead of limits

Most AI onboarding teaches users what the feature can do. Tooltips list capabilities, the empty state shows three example prompts, and a carousel explains the underlying model. Then the user tries their own case, which is not one of the three, and gets a mediocre result.

A better onboarding teaches the boundary. Show two examples that work and one that doesn't, and explain why. Let the user paste their own sample during setup and see the result before committing. If the sample fails, that's a successful onboarding: the user learned the limit at zero cost.

This is uncomfortable for product teams because it foregrounds failure. But the alternative is worse. A user who discovers a limit in production, during a deadline, doesn't just distrust that output. They distrust every output that came before it, because they no longer know which ones were inside the boundary.

The same logic applies to documentation. A page titled "What this assistant is good at" is marketing. A page titled "When to double-check the extraction" is a support tool that reduces both errors and tickets.

Patterns that destroy trust, and what to do instead

Pattern	Why it backfires	Alternative
"AI-powered" as the main value proposition	Frames the feature as magic, invites scrutiny of the mechanism instead of the outcome	Describe the task and the limit: "extracts totals from these invoice types"
A single accuracy number	Hides segment failures and encourages overgeneralization	Per-segment error rates in the user's units, plus known failure cases
Auto-apply on first use	Asks for trust before evidence	Read-only preview, then staged writes
Model-generated explanations	Plausible text is not evidence	Source receipts, field-level provenance, excluded inputs
All-or-nothing accept	One error invalidates the batch	Per-item accept, edit, reject
Confidence percentages	False precision, anchoring	Categorical uncertainty plus a review action
Hidden fallback to a cheaper model	Silent quality change	Disclose degradation, mark affected outputs
Opt-out buried in settings	Feels coercive	Per-feature and per-category controls in context
"Just review everything"	Moves the work, doesn't reduce it	Prioritize review by risk and uncertainty

Measure verification, not satisfaction

If you ask users whether they trust the feature, you'll get an answer shaped by politeness and recency. Measure behavior instead.

Verification time: how long between output appearing and the user acting on it. If it's near zero on high-stakes items, they're either over-trusting or not using it.
Edit rate by field: which parts of the output get corrected. This is your best product roadmap.
Rejection rate by segment: where the feature fails.
Opt-out rate: who turns it off and when. A spike after a model change is a signal.
Manual re-keying: the behavior in the opening story. Detect it by comparing downstream data entry with the feature's output; if users are re-entering values elsewhere, the feature is theater.
Trust calibration: pair a short survey with an objective check. "How confident are you in this output?" followed by "was it correct?" lets you see whether confidence tracks accuracy. A well-calibrated "not sure" is a success, not a failure.
Support contacts per thousand uses: a rising ratio after launch usually means the feature is producing plausible output that users can't verify, which is the worst combination.

The last metric is the one most teams miss. A user who says "I don't trust it for scanned invoices, but I do for digital ones" is exactly the outcome you want. A user who says "I trust it" across the board is a risk.

Recovering trust after a visible failure

Sooner or later the feature will be wrong in a way the user notices. How you handle that moment matters more than a year of good average performance.

What helps:

Acknowledge the specific failure, not a generic apology.
Show what the system will do differently, if anything. If the answer is "nothing," say that.
Make the correction permanent where possible.
Give the user a way to report the case into the evaluation set, and tell them when it's fixed.

What doesn't help: a banner claiming the model has improved, a survey asking how they feel, or a support response explaining that AI is probabilistic. The user already knows it's probabilistic. They want to know whether their specific case will recur.

What not to promise

Skeptical users have long memories for promises that didn't hold. Three promises are especially costly.

"It will never make a mistake." It will. Every system does. The promise converts an ordinary error into evidence of dishonesty, and the user re-evaluates everything you've told them.

"You don't need to check it." This is the opposite of what a consequential workflow needs, and it's the fastest route to automation bias. Ask for appropriate verification instead, and make that verification cheap.

"It learns your workflow automatically." If true, show it. If it learns from corrections but only within a session, say that. Users who expect adaptation and don't see it conclude the feature is static; users who are told it's static and see adaptation are pleasantly surprised.

The honest version is narrower and more persuasive: "It handles these cases well. It will flag the ones it can't. You approve anything that writes." That sentence is a contract, and a feature that keeps it earns more trust than one that overpromises and mostly delivers.

A design checklist for skeptical users

Before shipping, answer these from the user's chair.

Can a new user complete the task without accepting any AI output?
Is the first interaction read-only, or does it write before evidence exists?
Does the UI state what the feature does badly, in the user's vocabulary?
Can the user check a claim against a source in under a minute?
Is acceptance per item, and is correction faster than manual entry?
Does a failure preserve work and resume rather than restart?
Is there an in-context way to turn the feature off for one category or client?
Are model changes announced where the user will see them?
Is uncertainty expressed as an action, not a fake probability?
Does the product measure verification behavior, not just satisfaction?

A feature that passes this list doesn't require users to trust AI. It lets them use it while trusting their own judgment, which is the only kind of trust you can honestly ask for. The users who check your work are not a barrier to adoption. They're the QA process you didn't have to hire, and the design that supports them is the same design that survives a bad model day.