How to roll out an AI feature to 1% of users safely

“We’ll enable it for one percent of users and watch what happens” sounds cautious. It is not a safety plan by itself.

One percent can still include the wrong users, the most sensitive data, a noisy tenant, or the first request that triggers a costly loop. It can also be too small to reveal a quality regression, especially when the affected behavior is rare. A canary reduces exposure; it does not replace evaluation, access controls, or a rollback path.

Rolling out an AI feature safely means controlling who sees it, what actions it can take, what evidence is collected, and how the team responds when a signal crosses a threshold. The percentage is only one parameter. This article lays out a staged rollout from offline evaluation to a one-percent cohort, with concrete gates for quality, cost, latency, and safety.

Decide what the canary is meant to learn

Before selecting the cohort, write down the hypotheses the canary should test. Examples:

Does the new prompt reduce incorrect routing without increasing escalations?
Does the feature remain within the latency budget on real account data?
Are users able to tell a draft from a completed action?
Does retrieval stay inside the user’s tenant and permission scope?
Does cost per successful task fit the plan’s economics?

If the team cannot state what it is measuring, the canary becomes a demo with production traffic. The user sample may be small, but the blast radius is still real.

Define a baseline before rollout. Compare against the current workflow, not an empty dashboard. Record current task completion, escalation, correction, latency, and cost rates. For a new feature with no baseline, choose proxy measures and document their limitations.

Set three kinds of criteria:

Hard stop conditions: unauthorized data access, an unintended write, unsafe external action, or a runaway cost event.
Quality gates: minimum task-specific pass rates and maximum correction or escalation rates.
Service gates: latency, error, queue age, and cost limits.

The exact values depend on the feature and should be agreed before the canary starts. Avoid choosing thresholds after seeing results; that makes it too easy to redefine success around whatever happened.

1% is a cohort design problem

A percentage alone does not tell you which users receive the feature. Use a stable assignment so the same user or tenant does not move in and out of the experience between requests.

For a multi-tenant product, tenant-level assignment is often safer than user-level assignment. If one organization’s users share records or workflows, splitting them across old and new behavior can create inconsistent outputs and complicate support. For a purely personal feature, user-level assignment may make sense.

Avoid selecting the first 1% by signup order or by a random list without considering the risk. The cohort should be representative enough to learn from, while excluding users for whom the feature is not yet approved or supported. Consider language, region, accessibility needs, usage volume, data sensitivity, and tenant size. Do not accidentally choose only internal power users and conclude the general experience is ready.

A stable hash assignment can keep users in the same cohort without storing a list of every request:

import hashlib

def cohort_bucket(subject_id: str, rollout_key: str) -> int:
    material = f"{rollout_key}:{subject_id}".encode("utf-8")
    digest = hashlib.sha256(material).digest()
    return int.from_bytes(digest[:8], "big") % 10_000

def enabled_for_subject(subject_id: str, rollout_key: str, percent: float) -> bool:
    threshold = int(percent * 100)
    return cohort_bucket(subject_id, rollout_key) < threshold

This is an illustration of deterministic assignment, not a feature-flag SDK. The rollout key should change when you intentionally create a new experiment; the subject ID should be an opaque internal identifier. Store the resolved assignment with the operation so support can tell which version a user saw.

Keep the flag evaluation server-side when it controls data access or side effects. A client-side flag may hide a button, but it is not authorization.

Separate exposure from execution

An AI feature can be rolled out in several dimensions:

who can see the interface;
which model or prompt version is used;
whether the system generates a suggestion;
whether a suggestion is applied automatically;
which tools are available;
how much traffic is sampled for evaluation.

Do not bundle all of those into one boolean called ai_enabled. For a new agent with write tools, the safest canary may allow users to see proposals while keeping execution disabled. That lets the team measure task quality without exposing live side effects.

Use separate controls for:

feature visibility;
model/prompt version;
tool permissions;
write execution;
traffic sampling;
emergency shutdown.

The emergency kill switch should be independent of the application deploy. It should disable new model calls or side effects without waiting for a build. For a write-capable agent, the write kill switch should be separate from the read-only assistant so users can still access harmless explanations while execution is paused.

Shadow before visible rollout when possible

If the current system or a human workflow already produces a result, run the new model in shadow mode first. Generate proposals on real inputs but do not show them to users or execute tools. Compare the proposals with actual outcomes and inspect disagreements.

Shadow mode answers different questions from a user-facing canary:

Is the model capable of proposing useful results?
Does it choose the expected tool and arguments?
Are there segments where it systematically fails?
What is the estimated cost and latency under realistic traffic?

It does not answer whether users understand the UI, whether they trust the result, or whether they will accept the workflow. Those require visible exposure. It also does not prove the live write path is safe; test that separately in a sandbox and behind authorization controls.

Avoid shadowing every request indefinitely. Shadow traffic still consumes model capacity, may send sensitive data to a provider, and can create a second processing path that needs retention and observability controls. Sample according to risk and volume, and make the data handling part of the rollout plan.

Evaluation before traffic

The one-percent cohort should not be the first time anyone checks the behavior. Before rollout, run:

deterministic tests for permissions, input validation, schemas, and tool behavior;
a versioned eval set with typical, ambiguous, adversarial, and high-risk cases;
slice-level analysis by intent, language, tenant type, and input length;
repeated runs for high-impact nondeterministic behavior;
latency and cost tests using production-like payload sizes;
fault injection for timeouts, rate limits, empty retrieval, and partial tool failures.

OpenAI’s evaluation guidance emphasizes task-specific data, continuous evaluation, and human calibration rather than relying on generic metrics. Treat those evaluations as the gate before the canary, not as a replacement for live monitoring. (OpenAI, evaluation best practices)

The eval report should include denominators and examples, not only percentages. A high aggregate score can hide a failure on one important slice. The launch owner should inspect the cases where the candidate differs from the baseline, especially if the change affects a tool call or a factual claim.

Measure outcomes at the task level

For a one-percent rollout, instrument both exposure and outcome. Useful fields include:

rollout group and feature version;
prompt, model, and tool-schema versions;
operation ID and tenant or user reference;
task completion and escalation status;
user edits, overrides, cancellations, and repeat attempts;
latency by stage, not just end-to-end;
token usage and cost per successful task;
error class, retries, and unknown outcomes;
policy denials and any side effect performed.

Do not log raw user content by default just to compare cohorts. Use content minimization, protected traces, and explicit retention rules. If content review is needed, sample and route it under the product’s privacy controls.

Define a primary metric and guardrail metrics. For a support assistant, the primary metric might be issue resolution without a correction or repeat contact. Guardrails might include escalation rate, policy violations, latency, and cost. “Users clicked the AI button” is adoption, not task success.

One percent can be too small to measure quality

A small canary is useful for containing exposure, but it may not provide enough observations to estimate rare failure rates. If a harmful action occurs once per 10,000 requests, a 1% cohort over a short period may see no examples. “We saw no failures” is not evidence that the failure cannot happen.

Use the canary to validate operational behavior and catch gross regressions. Use offline evals, simulation, and targeted adversarial testing for rare cases. Report the number of tasks and the observation window alongside the metric. If a metric is based on 20 interactions, do not present it as a stable production rate.

For an experiment intended to compare outcomes, consider statistical power and the cost of exposure. A one-percent release can be the first stage, followed by a larger randomized cohort if the hard stops remain clear. Do not ramp simply because the calendar says the canary has run for a week; ramp when the evidence is sufficient for the decision you are making.

Set stop and rollback conditions before launch

The rollout plan should answer:

Which signals trigger an immediate pause?
Who is authorized to pause the feature?
Can the feature be disabled for one tenant or only globally?
What happens to in-flight tasks when the flag changes?
How are pending writes reconciled?
Where are traces and runbooks available?
How is the cohort notified if behavior changes?

Hard stops should not require a debate during an incident. Examples include confirmed cross-tenant data exposure, an unauthorized side effect, repeated duplicate writes, or spending beyond a configured cap. A quality regression may need investigation; a permission breach should trigger the kill switch.

Rollback must cover data as well as code. If the AI feature wrote incorrect records, reverting the frontend does not undo those writes. Have a reconciliation or compensation plan for any mutable data, and preserve an audit trail of actions already taken.

Expand gradually, not mechanically

A rollout can progress from internal users to a small external cohort, then a larger cohort, but each stage should answer a new question. A simple progression might be:

Internal staff, read-only suggestions.
Small external cohort, suggestions visible but not automatically applied.
Broader cohort, low-risk reversible actions enabled.
Higher-impact actions only after separate evaluation and approval design.

The exact percentages are not universal. A “1%, 5%, 25%, 100%” sequence is a mechanism, not a safety argument. Cohort size should reflect traffic volume, risk, support capacity, and the metrics you need to observe.

Keep the assignment stable as the cohort grows. If the denominator changes unexpectedly, users can shift groups and contaminate the comparison. Record each ramp event and the decision evidence that justified it.

Choose a cohort that tests the actual risk

Random assignment is valuable when the goal is to compare outcomes, but a random 1% can still be a poor safety cohort. If the feature affects billing administrators, a cohort with no billing administrators tests nothing about the write path. If the product supports several languages, a cohort containing only English requests will not reveal localization failures.

Use a stratified rollout where risk requires it. Keep the cohort small within each relevant segment, or launch separate canaries for distinct workflows. Do not combine highly different actions—summarizing a public help page and changing account permissions—under the same rollout flag and then report one blended result.

Avoid deliberately selecting only “easy” users. Internal staff and friendly design partners can help discover usability issues, but they may know how to work around model errors and may not represent production data. Use them as one stage, then include an external cohort with support coverage and explicit limits.

For multi-tenant products, decide whether the experimental unit is a user, workspace, or tenant. If users collaborate on shared records, tenant-level assignment avoids one person seeing an AI-generated value while a colleague sees the old workflow. For cross-tenant services, stable assignment and server-side region/permission checks must work together; an experiment flag must not move data into a different processing path.

Separate rollout monitoring from causal measurement

A canary has two jobs: detect operational harm quickly and estimate whether the feature improves outcomes. These require different evidence.

Operational monitoring should be continuous and fast: error rate, p95 latency, spend, queue age, policy denials, and unexpected side effects. A hard stop can trigger after one confirmed security boundary violation.

Outcome measurement usually needs a comparison group and a longer window. Users may take time to correct a draft, reopen a ticket, or report a wrong answer. Compare cohorts with the same eligibility criteria and track the same task-level outcome. If the rollout changes the workflow for one segment while product policy changes for everyone, record that change so it is not mistaken for an AI effect.

Avoid optional participation bias where only enthusiastic users turn on the feature and are then compared with all other users. Their workflows may differ before the feature is enabled. If random assignment is not possible, describe the selection bias and treat results as directional rather than causal.

Keep a small holdout only if it is ethically and operationally appropriate for the product. Do not withhold a safety improvement from users merely to preserve an experiment. In that case, use pre/post analysis, shadow comparison, or another design reviewed by the product and risk owners.

Prepare support and on-call for the canary

The first user who sees a broken answer is often not an engineer. Support needs a safe way to identify whether the feature was enabled, which version ran, and what state the operation reached. Provide a correlation ID or task reference that does not expose raw prompts in a support dashboard.

Before enabling the cohort, give support:

the user-facing behavior that is expected;
examples of partial, queued, refused, and failed states;
a way to look up prompt/model version and tool outcome;
instructions for preserving user work and escalating incidents;
a clear statement about what support can and cannot safely retry;
the owner and channel for pausing the rollout.

On-call should have a one-page runbook with the stop conditions, relevant dashboards, kill switch, and steps for reconciling uncertain writes. If the first response to “the AI changed this record” is “try again,” the rollout is not ready.

Define what happens to work already in flight

Turning off a feature flag prevents new assignments; it does not automatically stop tasks that already started. A queued job may still execute a tool call after the UI has been disabled. A streaming request may still be running. A human approval item may remain open after the model version is rolled back.

For each task type, specify the kill-switch semantics:

stop new requests immediately;
cancel safe in-flight model calls where cancellation is supported;
prevent pending writes from executing without current authorization;
leave already completed actions intact and reconcile them;
mark incomplete tasks as paused, failed, or awaiting review;
preserve the user’s input and make the state visible.

Do not attempt to undo every completed action automatically. Some actions require compensating operations or human judgment. The kill switch should stop further harm; the recovery workflow should determine what to do about effects that already happened.

Include the flag’s behavior in the task state machine and test it while a tool call is pending. A flag that hides a button but leaves a worker with write capability is a UI switch, not an operational kill switch.

Define how a canary failure changes the rollout

Not every bad signal requires the same response. Map signals to actions before launch:

a single confirmed cross-tenant access or unauthorized write pauses the relevant capability immediately;
a provider outage may pause new requests while leaving read-only features available;
elevated latency may move work to a queue or reduce the cohort;
a rising correction rate may hold the ramp while the team reviews examples;
a spend-cap breach should stop new paid operations rather than merely page finance;
a small sample with no observed incidents should keep the rollout at its current stage until more evidence exists.

The response should be scoped. If one tool is unsafe, disable that tool while preserving safe search or drafting. If one tenant’s configuration is broken, disable the tenant rather than every user. However, scoped containment must not preserve a path to the same harmful action through a fallback tool.

Record who made the decision, which signal triggered it, and which cohort or capability changed. This helps the team distinguish a deliberate pause from an unexplained drop in traffic and makes it possible to resume after the root cause is fixed.

Tell users what the canary changes

Internal rollout controls should not create a confusing external experience. Users may see different behavior across workspaces or over time. Explain when AI-generated content is a draft, when it is automatically applied, and how to report an incorrect result. If the feature processes data through a new provider or changes retention behavior, make sure the privacy notice and product settings are accurate before enabling the cohort.

Avoid using a canary as a way to quietly test a high-impact action on unsuspecting users. For sensitive decisions, use shadow mode or explicit approval first. A limited percentage reduces exposure, but it does not remove the obligation to provide a clear user-facing contract.

Support and customer success should know which users are in the cohort and which version they received. The assignment should remain stable enough that a support agent can reproduce the experience, while still allowing the product owner to remove a tenant quickly if a configuration issue appears.

Common rollout mistakes

Mistake	Why it fails	Better approach
Treating 1% as automatically safe	A small group can still encounter severe harm	Define action scope and hard stops
Enabling write tools with the UI	Exposure and execution risk are different	Separate visibility and execution flags
Measuring only engagement	Clicks do not prove task value	Measure accepted, successful outcomes
Using one aggregate quality score	Hides a failing slice	Segment by task, tenant, language, and risk
Rolling out by elapsed time	Time does not equal evidence	Require predeclared gates
No per-tenant kill switch	One tenant’s data shape can harm all users	Support scoped disablement
Shadowing all traffic forever	Adds cost and data processing without a decision	Sample and set an exit criterion
Rolling back code only	Existing writes remain	Plan data reconciliation and compensation

A canary checklist

Before enabling the first external cohort, confirm:

Offline evaluation and deterministic security tests passed.
The cohort is stable and selected server-side.
The feature’s write permissions are separately controlled.
Tenant, region, and data-retention constraints are enforced.
Cost and latency budgets have alerts and caps.
Tool calls are authorized and idempotent where needed.
The UI distinguishes draft, partial, queued, failed, and completed states.
Sampling and logs are privacy-reviewed.
The on-call owner and support path are known.
Kill switches and rollback were tested in staging.
Hard-stop conditions and ramp criteria were agreed in advance.
The team has a plan for handling in-flight and already-completed actions.

A 1% rollout is safest when it is one step in a controlled evidence plan. Choose a cohort that can teach you something, limit what the system can do, observe task outcomes and failure modes, and stop automatically when a hard boundary is crossed. Then expand because the evidence supports it—not because the first hundred users happened to be quiet.