Week one, the feature is magic. People paste in their messiest document, screenshot the output, and send it to a colleague. Session counts spike. Someone in the company Slack channel calls it "genuinely useful" and the team puts it on the roadmap slide.
Week nine, the same people use it three times a day and complain that the third paragraph is always a bit generic.
Nothing about the model changed. What changed is the frame. In week one, users were evaluating whether the feature could do the thing. By week nine, they are evaluating whether it does the thing reliably enough to stop thinking about it. Those are different tests, and a feature designed to pass the first one often fails the second.
This is the novelty cliff, and it is not a marketing problem. It is a product and interface problem with specific, testable symptoms and specific design responses.
The novelty budget, and why it expires
A new AI feature gets a temporary allowance. Users forgive latency, tolerate awkward phrasing, re-read output twice, and work around missing controls, because the capability itself is interesting. That allowance is the novelty budget.
It is spent by repetition. The tenth time someone asks for a summary, they are no longer impressed that a summary exists. They are timing it. The twentieth time, they notice it always opens with "This document outlines." The thirtieth time, they have stopped reading the first sentence entirely and are scanning for the number they need.
Three phases are worth distinguishing, because each has a different failure mode:
| Phase | User question | What they tolerate | What breaks the experience |
|---|---|---|---|
| Curiosity (days 1–7) | "What can this do?" | Slow responses, odd formatting, dead ends | Nothing much; they are exploring |
| Evaluation (weeks 2–6) | "Is this better than doing it myself?" | Occasional errors if the happy path is good | A wrong answer they almost shipped |
| Integration (week 7+) | "Can I stop paying attention?" | Very little | Unpredictability, re-work, silent failure |
The integration phase is where most products lose. In that phase the user has folded the feature into a workflow. They are not looking at the output closely; they are looking at it just enough. That is the highest-trust state a feature can reach, and it is also the state in which a single confident wrong answer does the most damage, because the user's attention is now allocated elsewhere.
Designing for integration means designing for a user who is half-watching. That is a very different brief from designing for a user who is curious.
Expectation one: predictable shape, not maximum capability
Ask users what they want from a mature AI feature and they will say "better answers." Watch what actually frustrates them and it is usually inconsistency: the summary is bullet points on Monday and prose on Wednesday, the extracted date is sometimes ISO and sometimes "next Friday," the tone shifts between formal and chatty for no reason they can identify.
Users build a mental model of output shape within about five uses. Once that model exists, deviations read as bugs even when the content is better. A model upgrade that improves factuality by four points but changes the formatting will generate more complaints than praise.
The practical implication is that output shape should be a product contract, not a model behavior. If a summary has three sections, enforce three sections. If a date field is ISO 8601, validate it and retry on violation rather than hoping. If the feature always ends with a suggested next step, make that structural.
from pydantic import BaseModel, Field
from datetime import date
class TicketSummary(BaseModel):
problem: str = Field(max_length=400)
customer_impact: str = Field(max_length=300)
requested_action: str = Field(max_length=300)
deadline: date | None
confidence: float = Field(ge=0.0, le=1.0)
A schema like this does two things beyond validation. It makes the shape stable across model versions, and it gives you something to diff when you change a prompt. "The summary got worse" is unactionable. "The requested_action field went from 92% populated to 71%" is a bug report.
There is a cost. Strict schemas reduce the model's freedom to be helpful in unexpected ways. For a mature feature that is usually the right trade. For a feature still in the curiosity phase, it may not be. The mistake is shipping the exploratory version and never tightening it.
Expectation two: fast enough to keep my place
Novelty-era users wait. They watch the tokens stream in, because watching is part of the experience. Mature users tab away.
The threshold is not absolute latency; it is whether the wait costs them their train of thought. A four-second response while they are reading the prompt is fine. A four-second response in the middle of a rapid sequence of edits is an interruption, because they have to reload context in their head when it returns.
This is why "average latency" is a misleading metric for a mature AI feature and p95 in-context latency is not. The moments that matter are the ones where the user is mid-flow: accepting a suggestion, correcting a field, asking a follow-up. Those are fast interactions or they are not.
Three responses work better than raw speed:
Keep the user's place. If a suggestion takes six seconds, do not block the editor. Let them keep typing, show the suggestion when it lands, and make accepting it a single keystroke. The wait still exists; it just stops being an interruption.
Show partial value early. A first paragraph in 900ms changes the perceived cost of a nine-second total. This is not deception if the partial output is genuinely usable and clearly incomplete. It is deception if you animate a fake progress bar.
Make repetition free. The most common mature-use pattern is running the same operation on many items. If each one takes four seconds and there is no batch path, the feature becomes unusable at scale even though it is pleasant for one item. Users will build their own workaround, which is usually a spreadsheet.
That last point is worth dwelling on. Novelty users do the operation once, on the most interesting input. Mature users do it forty times, on the boring ones. A feature optimized for the interesting case and never tested on the boring case will feel excellent in a demo and tedious in a Tuesday afternoon.
Expectation three: let me fix it, not re-roll it
The single biggest interface difference between a novelty feature and an integrated one is the correction mechanism.
In the novelty phase, "regenerate" is fine. The user is exploring; a different draft is interesting. In the integration phase, regenerate is hostile, because it throws away the parts that were right. If a generated email has a perfect opening and a wrong closing, the user does not want a new email. They want to edit the closing.
Mature expectations cluster around four affordances:
- Direct editing. The output is text in a field, not a rendered block. Users will fix a word rather than ask for a rewrite.
- Partial acceptance. Keep this paragraph, discard that one. Especially for lists, tables, and multi-section documents.
- Targeted regeneration. "Rewrite this paragraph, shorter" beats "try again."
- Undo that is actually undo. Reverting to the previous version, not re-running the generation and hoping for something similar.
A useful test: can a user take a generated output that is 80% correct and reach 100% in less time than writing it from scratch? If the answer is no, the feature is a novelty regardless of how good the raw output is. The metric that captures this is edit distance — how many characters the user changes before accepting — combined with time-to-accept.
🔍 Why this matters: edit distance is one of the few AI metrics that is both a quality signal and a usability signal. A rising edit distance means either the model got worse or the output became harder to correct. Those need different fixes, and you can distinguish them by looking at where the edits happen.
Expectation four: remember the boring things, forget the creepy ones
After a few weeks, users stop restating context. They expect the feature to know that their team calls the product by an internal name, that their documents use British spelling, that a "brief" in their org means one page and not five.
That expectation is reasonable and it is also a trap, because the same mechanism that remembers preferences can remember something a user did not intend to persist. The line users draw is fairly consistent, even if they cannot articulate it:
| Users expect remembered | Users find unsettling |
|---|---|
| Formatting and tone preferences | Inferred personality traits |
| Vocabulary and internal names | Anything derived from another person's data |
| Recent task context within a session | Context from a different product surface |
| Explicitly stated instructions | Guesses about intent the user never confirmed |
| Their own past corrections | Corrections made by a colleague without attribution |
The design rule that satisfies both columns is: remember what the user did, not what you inferred. A correction is an action. "User prefers shorter summaries" is an inference, and it should be shown, editable, and attributable — or not stored at all.
There is also a freshness expectation. A preference expressed eight months ago in a different role should not silently govern today's output. Memory that never expires will eventually be wrong, and wrong memory is worse than no memory because it produces confident, personalized errors.
Expectation five: know where you end
Mature users want a clear boundary. They want to know which parts of the product the AI handles, which parts it merely assists, and which parts it will not touch.
A feature that sometimes summarizes a document, sometimes answers a question about it, and sometimes edits it has no boundary, and users respond by not trusting any of the three modes. They cannot predict what will happen when they press the button, so they press it cautiously or not at all.
The fix is usually interface, not capability. Separate entry points for separate intents. Labels that say what will happen. A scope statement in the empty state that answers "what is this for" in one sentence, phrased in the user's vocabulary rather than the model's.
Refusal is part of this. A feature that says "I can't answer questions about contracts in this tool; use the contract review workflow" is more trustworthy than one that attempts an answer badly. The refusal signals that the boundary is real. Users forgive a clear no far more readily than a vague maybe.
Expectation six: consistency with the rest of the product
Users do not experience the AI feature in isolation. They experience it next to a form, a table, a permission model, and an undo stack that already have conventions.
When the AI feature violates those conventions, the violation is what they notice. If everything else in the product saves automatically, the AI panel that requires a "Save" click feels broken. If permissions elsewhere are enforced by hiding actions, an AI action that fails with a 403 after the fact feels unsafe. If the rest of the app uses sentence case, a chat interface that shouts in title case reads as a different product.
The consistency checklist is boring and effective:
- Same vocabulary for the same objects. If the app calls them "cases," the AI does not call them "tickets."
- Same permission semantics. If the user cannot see a record, the AI cannot summarize it, and the error should look like the app's normal permission error.
- Same save model. Autosave, draft, or explicit save — pick the app's convention.
- Same undo semantics. If undo is a toast, the AI's undo is a toast.
- Same loading language. If the app says "Loading," the AI does not say "Thinking."
None of these affect model quality. All of them affect whether the feature feels like part of the product or an experiment bolted onto it. In the integration phase, that feeling is most of the experience.
The trust curve turns, and then it inverts
There is a second-order effect that catches teams by surprise. In the curiosity phase, the risk is under-trust: users do not believe the output and check everything. In the integration phase, the risk inverts. Users stop checking.
This is ordinary automation bias, and it is stronger with fluent output. A bulleted summary in confident prose reads like a finished artifact. The user's attention moves from "is this right" to "where do I send it." By the time a wrong number reaches a customer, several people have handled the document and none of them read the sentence that mattered.
The design response is not to make the output less trustworthy. It is to place friction where the cost of error is high, and only there. A few patterns work:
- Provenance on specific claims. Not a citation for every sentence, which users ignore, but a link on the numbers and named entities. The parts people actually copy.
- Confidence shown as behavior, not a number. A score of 0.72 means nothing to a user. "I couldn't find this in the source document" means something.
- A confirmation step for irreversible actions. Sending, charging, deleting, publishing. The AI can draft; the user commits.
- Diff view for edits to existing content. Showing what changed is how a half-watching user catches a silent rewrite of a clause they cared about.
The failure mode to avoid is symmetric friction: forcing confirmation on everything, which trains users to click through confirmations without reading. Friction only works where it is rare.
🚨 Production warning: once users stop reading, your error messages stop working. A toast that says "some content may be inaccurate" is invisible after week three. If accuracy matters, the warning has to be attached to the specific claim, at the moment of use, not to the feature in general.
Onboarding for week nine, not week one
Most AI features are onboarded for the curiosity phase: a tour, a few example prompts, a suggestion to "try asking it to summarize something." That onboarding teaches exploration. It does nothing for the user who will use the feature forty times next month.
Steady-state onboarding is about installing a habit and a boundary. It answers three questions in the first thirty seconds of real use:
- When should I use this instead of doing it myself? A concrete task, not a capability list. "Use this when a ticket has more than three replies and you need the customer's actual ask" beats "summarize anything."
- What will it not do? The boundary, stated plainly, before the user discovers it by failure.
- How do I correct it? The correction affordance, demonstrated once. Users who never learn that a paragraph can be edited in place will keep regenerating the whole document.
The third one is the most commonly skipped and the most valuable. A single tooltip on the first generated output — "click any paragraph to edit it before accepting" — changes the correction behavior of a user permanently.
Progressive disclosure matters here too. A mature feature should not show every control on first use, but it should not hide the important ones forever. The pattern that works is: defaults that are right for most users, plus a visible path to the controls that power users need, discovered at the moment they need them rather than in a settings page.
The metrics that survive the novelty phase
Usage metrics are designed for the curiosity phase and they lie in the integration phase. Session count, messages per user, and time-in-feature all go up when a feature is novel and interesting, and they can stay flat or rise while the feature becomes less useful.
Metrics that hold up:
| Metric | What it measures | Why it survives |
|---|---|---|
| Weekly active tasks completed | Real work done with the feature | Not inflated by exploration |
| Time to accept | Whether output is usable as-is | Falls when quality improves |
| Edit distance before accept | Correction burden | Rises when output is subtly wrong |
| Regeneration rate | Whether the first attempt lands | High rates signal unpredictability |
| Abandonment after first output | Whether users give up | The clearest failure signal |
| Escalation to human | Whether the feature knows its limits | A healthy feature escalates sometimes |
| 30-day retention of use | Whether it became a habit | The single best integration proxy |
The pair worth watching together is acceptance rate and edit distance. High acceptance with high edit distance means users are taking the output and fixing a lot of it — the feature is helping, but not as much as the acceptance metric suggests. Low acceptance with low edit distance means users reject outputs that were nearly right, which is usually an interface problem rather than a model problem.
Add time-to-accept to that pair and you get a crude but honest picture of whether the feature is saving time or merely relocating it. A workflow where acceptance is high, edit distance is low, and time-to-accept is unchanged has not automated anything. It has moved the work from writing to reviewing, and reviewing is still work.
Demo debt, and how to pay it down
Most AI features accumulate demo debt in the first month: decisions made to make the feature impressive in a five-minute walkthrough, which become liabilities at daily use.
The classic forms:
- A chat box as the only interface. Great for demos because anything can be typed. Terrible at scale because nothing is discoverable and every task requires composing a prompt.
- Unbounded output. Long answers look comprehensive in a demo and are unreadable on the fourth use.
- No empty state. The demo starts with an interesting question already typed. Real users see a blank box and a cursor.
- Happy-path error handling. Demos do not hit rate limits.
- Novel interactions for their own sake. A custom streaming animation instead of the product's normal progress pattern.
Paying it down is unglamorous. Add the structured entry points. Cap the output. Write the empty state. Add the error states. Replace the custom animation with the design system's spinner. Each change makes the feature slightly less impressive in a demo and meaningfully better on the ninth Tuesday.
💡 Practical note: the cheapest way to find demo debt is to sit with a real user for thirty minutes of their actual work. Not a usability test with tasks you wrote. Their queue, their documents, their interruptions. The gap between that session and your demo script is the debt.
What to build differently, in order
If a feature is past its novelty phase and underperforming, the fixes are rarely "a better model." They are usually in this order:
- Make the output shape deterministic. Schema it, validate it, retry on violation. Stability beats marginal quality.
- Add direct editing and partial acceptance. Reduce the cost of being 80% right.
- Put the feature where the work happens. Inline in the editor, in the row, in the composer — not in a separate panel the user has to remember to open.
- Add a batch path. Mature use is repetitive. One-at-a-time is a demo pattern.
- Make memory visible and editable. Show what it remembers. Let users delete it.
- Write honest empty, loading, and error states. Especially the error states.
- Instrument acceptance, edit distance, and abandonment. Stop optimizing sessions.
- Only then consider a model upgrade, and evaluate it against the metrics from step 7.
The ordering matters because steps 1 through 6 change what "better" means. Running a model comparison before you have stable output shapes and a correction mechanism will tell you which model produces nicer prose, not which one makes the feature more useful.
Two tempting fixes that make it worse
When a feature stops feeling useful, two responses are common and both backfire.
The first is adding more capability. If summaries are underwhelming, add question answering. If question answering is underwhelming, add document drafting. The feature becomes a general assistant with no clear job, and users respond by using it for none of them. Scope creep reads as ambition internally and as incoherence externally. A mature feature usually needs less surface area, not more.
The second is adding personality. When output feels flat, teams add a warmer voice, emoji, or conversational flourishes. Users in the integration phase are not looking for a companion; they are looking for a competent tool that does not waste their attention. Personality that was charming in week one becomes noise by week nine, and it is noise that has to be edited out of every output before it can be shipped to a customer.
The exception is when the personality is the product — a creative writing tool, a tutoring interface, a consumer companion. In those cases the voice is the value, and the expectation curve is different. For a work tool embedded in someone's queue, competence reads better than charm.
The test worth applying
A simple question separates a feature that has survived the novelty phase from one that has not: if you removed the AI and replaced it with the previous manual process, how many users would notice within a week, and how many would be annoyed?
Curiosity-phase features fail this test. Users would notice, because the feature is visible, but they would not be annoyed, because the manual process was never much worse. Integration-phase features pass it, because the AI is now load-bearing in a workflow and its absence costs real time.
Getting from one to the other is not about making the model smarter. It is about making the feature boring: predictable in shape, fast in context, correctable, consistent with everything around it, honest about its limits, and quiet about its own cleverness. Boring is the goal. Boring is what "integrated" looks like from the inside.