Why your AI agent needs a dry-run mode

An agent says it is about to cancel a subscription. The operator clicks “test,” reads the trace, and sees a successful API response. The customer’s subscription is now cancelled.

That was not a dry run. It was a production write with a reassuring label.

A real dry-run mode lets an agent plan and validate an action without performing its external side effect. It is useful for debugging, shadow evaluation, permission reviews, prompt changes, and gradually introducing write-capable tools. But dry run is not just a flag passed to the model. It is a capability boundary in the application that must make writes impossible even when the model, prompt, or tool adapter behaves unexpectedly.

This article explains what dry run does and does not prove, how to implement it safely, how it differs from shadow mode and sandboxing, and how to turn dry-run results into evidence for a controlled rollout.

Dry run, shadow mode, simulation, and preview are different

These terms often get mixed together. They describe different behaviors:

Mode	What executes?	What it proves
Dry run	Model, retrieval, validation, and planning; external writes are suppressed	What the agent would propose under current inputs
Shadow mode	The agent proposes alongside the existing workflow; only the existing workflow acts	How proposals compare with current human/system decisions
Simulation	A fake or recorded environment handles reads and writes	How code behaves under a controlled world model
Preview	A user sees a proposed change and explicitly confirms it	Whether a human approves this exact action
Sandbox	Calls go to an isolated environment with test data	Whether integration behavior works outside production

A dry run may use live, read-only data to construct a proposal. A sandbox may still perform writes, but only in an isolated account. A preview is a user experience and may execute after approval. A shadow run usually compares against a parallel workflow. One does not substitute for the others.

The distinction matters for claims. “Dry run passed” does not mean the production write would succeed, that authorization would still be valid later, or that the user approved the action. It means the system produced a plan and the application prevented the real side effect during that run.

1. Put dry-run enforcement below the model

Do not rely on a prompt that says “do not make real changes in dry-run mode.” The model can misunderstand, the prompt can be omitted, or a new tool can ignore it. The application must enforce the mode at the layer that performs the side effect.

One simple design is an explicit execution context passed to every tool:

from enum import StrEnum
from dataclasses import dataclass
from typing import Any

class ExecutionMode(StrEnum):
    DRY_RUN = "dry_run"
    LIVE = "live"

@dataclass(frozen=True)
class ToolContext:
    mode: ExecutionMode
    operation_id: str
    actor_id: str
    tenant_id: str

def update_subscription(ctx: ToolContext, subscription_id: str, plan: str) -> dict[str, Any]:
    current = load_authorized_subscription(
        actor_id=ctx.actor_id,
        tenant_id=ctx.tenant_id,
        subscription_id=subscription_id,
    )
    validate_plan_transition(current, plan)

    if ctx.mode == ExecutionMode.DRY_RUN:
        return {
            "status": "would_update",
            "subscription_id": subscription_id,
            "from_plan": current.plan,
            "to_plan": plan,
            "side_effect_performed": False,
        }

    return subscription_service.change_plan(
        subscription_id=subscription_id,
        plan=plan,
        idempotency_key=ctx.operation_id,
    )

The tool still performs read authorization and validates the proposed transition. It skips only the side effect. Returning a structured would_update result makes the distinction visible to the orchestrator and UI.

In a production system, make it difficult for a tool author to forget the mode check. Wrap side-effecting services in a shared facade that requires an explicit execution capability. Add tests that assert dry-run contexts cannot reach the live client. A nullable boolean like dry_run=False is easy to omit; an enum or separate interface is harder to confuse.

2. Do not make a fake result look like a completed action

The most dangerous dry-run bug is returning a response that looks exactly like a live success. If the tool says {"status":"cancelled"} when it merely planned a cancellation, the agent may tell the user “Your subscription is cancelled.” Downstream steps may also continue under the false assumption that the change happened.

Use distinct statuses such as:

would_create, would_update, would_delete;
blocked_by_policy;
requires_approval;
live_action_succeeded;
live_action_failed;
outcome_unknown.

The output schema should include an explicit side_effect_performed field or a typed action result. The response renderer should use that field, not infer execution from a natural-language summary.

For an agent with multiple steps, dry-run mode must propagate to every child agent, queued task, callback, and tool adapter. A parent agent in dry run that invokes a live child workflow is not a dry run. Persist the mode with the operation, and reject any attempt to change it mid-run without an explicit user or operator action.

3. Separate planning from execution

A useful architecture has two phases:

Plan phase:
  read current state → propose actions → validate → produce a preview

Execute phase:
  re-authorize → re-read current state → verify approved proposal → execute

The plan should be a structured artifact, not just a paragraph in the trace. It can include the target resource, proposed fields, expected side effects, assumptions, policy checks, and an expiry. A reviewer can inspect it, compare it with current state, and approve or reject it.

Execution should not blindly replay the plan. Between planning and approval, the user may lose permission, the record may change, or the target may be deleted. Re-read and revalidate immediately before the write. If the plan no longer matches current state, invalidate it and ask for a new review.

This is a time-of-check/time-of-use boundary. A dry run that succeeds at 10:00 does not authorize a write at 10:30. It only shows that the proposal was valid against the state observed at 10:00.

4. Keep authorization active in dry-run mode

Dry run should suppress effects, not bypass controls. If a user lacks permission to view a record, the agent should not be able to use dry-run mode to inspect it. If a user cannot update a subscription, the dry-run result should say that the action would be denied.

There are two useful dry-run policies:

User-faithful dry run: evaluate permissions as the requesting user. This answers, “Could this user do this action right now?” It is the right mode for product previews and support workflows.

Privileged simulation: evaluate a hypothetical action using a dedicated test identity or policy fixture. This answers, “Would the workflow logic accept this input under these test permissions?” It belongs in tests and controlled operations, not in a user-facing preview.

Do not silently substitute a service account’s broad privileges for the user’s authority. The dry-run result should make the actor and scope explicit, and the live execution path should repeat the authorization check at the server-side tool boundary.

5. Record a plan that can be compared with the live result

For each dry-run operation, record enough structured metadata to compare the plan with what would actually happen:

stable operation_id and dry-run mode;
prompt, model, tool-schema, and policy versions;
actor and tenant references;
target resource IDs and observed versions;
proposed field changes or action parameters;
validation outcomes and reasons;
any human approval or edit;
whether the plan was later executed, rejected, or expired.

Avoid putting full sensitive tool arguments into general logs. For a shipping-address change, a broad operations trace may need the address reference and a hash, while a restricted audit record may hold the address under stricter access controls. Dry run does not make sensitive data safe to log.

When the live action occurs, link it to the plan but record the live result separately. That enables questions like “How often did the final action differ from the dry-run plan?” and “Which policy or state change caused a plan to be rejected?”

6. Dry run does not remove cost, latency, or data exposure

A dry run still executes the model, reads data, retrieves documents, and may call moderation or other services. It can cost nearly as much as a live operation. It may also send sensitive data to the same model endpoint. The only thing it suppresses is the side effect that the application has explicitly disabled.

If you shadow every production request with an agent run, account for the extra model traffic. Put a budget and sampling policy around the shadow path. Do not let a debugging mode multiply provider calls without a cap.

Be careful with user experience. If a user clicks a button labeled “Run,” then sees the assistant report a completed action during dry run, the product has lied even if the backend did nothing. Make the mode explicit in the UI and in the response: “Preview only—no changes made.” Do not silently turn dry-run output into a customer-facing claim of completion.

7. Use a real simulator for multi-step workflows

When a workflow has multiple dependencies, a boolean dry-run flag may not be enough. One tool can simulate a cancellation while another sends a live email. A simulator or recording adapter gives you a controlled set of reads and an isolated write sink.

class ToolAdapter:
    def __init__(self, read_client, write_client, mode, recorder):
        self.read_client = read_client
        self.write_client = write_client
        self.mode = mode
        self.recorder = recorder

    def call(self, name: str, arguments: dict):
        if name in READ_TOOLS:
            result = self.read_client.call(name, arguments)
            self.recorder.record(name, arguments, result, simulated=False)
            return result

        if name not in WRITE_TOOLS:
            raise ValueError(f"unregistered tool: {name}")

        if self.mode == ExecutionMode.DRY_RUN:
            result = simulate_write(name, arguments)
            self.recorder.record(name, arguments, result, simulated=True)
            return result

        result = self.write_client.call(name, arguments)
        self.recorder.record(name, arguments, result, simulated=False)
        return result

This sketch depends on explicit tool registries and a simulator for each write. The simulator must not call the live client, even to “check” whether the write would work. It can validate the payload against local policy and return an estimated result. If external service behavior matters, use the provider’s sandbox environment or a test double, not a hidden production request.

For multi-step flows, simulate intermediate state transitions in memory. If step one would create a draft and step two would send it, the simulator should make the draft visible to the second step without actually sending anything. This catches workflow logic errors that a list of independent “would call” results misses.

8. Compare dry-run proposals with human decisions carefully

Dry-run mode is useful for measuring what an agent would do before enabling writes. Compare its proposals with what competent operators actually did, but do not treat human behavior as unquestionable ground truth. Existing processes can be inconsistent, outdated, or subject to missing context.

For each case, label:

target correctness;
action correctness;
parameter correctness;
policy compliance;
whether the operator had additional context;
whether the human decision itself was later corrected.

Report disagreement by action type and severity. A 95 percent agreement rate across harmless labels cannot justify automating a payment action with one unexplained high-impact disagreement. Define promotion thresholds per action class, and include enough cases for the team to inspect the failure types—not just their frequency.

Dry-run comparisons also reveal process ambiguity. If two experienced reviewers disagree on whether an account should be suspended, the correct next step may be a policy decision, not another prompt revision.

9. Test the guarantee, not only the preview

Dry-run mode is a safety control, so it deserves tests at the boundary where side effects happen.

A dry-run context cannot invoke a live write client.
Every registered write tool has a dry-run implementation or is unavailable in that mode.
Child workflows inherit the parent’s mode.
A dry-run response cannot be parsed as a completed live action by downstream code.
The UI labels previews and preserves the original state.
Authorization still denies inaccessible records.
A stale plan is rejected before execution.
Retries in dry-run mode do not trigger live side effects.
A fault in the simulator fails closed rather than falling through to the live adapter.

The last test catches a particularly dangerous pattern: code that uses the simulator only when it is available, then falls back to the real client when a dependency is missing. In dry-run mode, simulator failure should stop the operation.

Run tests with instrumented clients that fail if any live write method is called. A test that only asserts the returned text says “preview” can pass while the external side effect has already happened.

Persist the mode as an operation attribute

Execution mode should be part of the operation record, not a transient request flag that disappears when a task enters a queue. Store it with the task ID, actor, tenant, and policy version. Every worker and child task should load the same mode from trusted server-side state.

Do not allow the model to request a transition from dry run to live. A user or authorized operator can approve a specific proposal through a separate endpoint. That endpoint should create an approval record bound to the proposal hash, target version, and approver identity. The live executor can then confirm that the approved proposal is exactly the one it is about to apply.

This prevents a subtle race: a user reviews one dry-run proposal, then the agent continues planning and changes the amount or target before execution. If the application only records “approved=true,” it may apply a different action from the one shown. Bind approval to immutable proposal data and invalidate it when that data changes.

If the application uses a queue, include the operation mode and proposal version in the job payload or a durable database reference. Never infer live mode because a queue worker lacks a dry-run field. Missing mode should be an error that fails closed.

Decide whether dry-run reads are live or simulated

Many dry runs use live, read-only data to make a realistic plan. That is useful, but it has consequences. The agent may see fresh personal data, trigger read-side audit events, consume provider quota, or create load on a production API. A “no writes” promise should not be misrepresented as “no production access.”

State which reads occur and who can request them. For a product preview, read through the same authorization filters as the live path. For a test harness, use a fixture or a snapshot. For a production shadow run, impose sampling, rate limits, and privacy controls so the second agent run does not double every request’s data exposure and cost.

Read-side effects exist too. A lookup may mark a notification as seen, update a “last accessed” timestamp, or trigger billing. Classify tools by actual behavior rather than by HTTP verb. If a nominal GET changes state, it is not a harmless read for dry-run purposes.

When a dry-run proposal is reviewed later, its source state may have changed. Include a timestamp and resource version in the preview. If the record is stale at approval time, regenerate the proposal instead of implying the preview is still current.

Measure whether dry run is helping

Dry-run mode has operational costs: model usage, read traffic, review time, and potential alert noise. Track whether it produces evidence that changes a rollout decision.

Useful measurements include:

proposals generated by action type;
invalid, denied, and incomplete proposals;
human acceptance, edits, and rejection reasons;
stale proposals at approval time;
simulator failures and attempted live-client access;
provider cost and latency attributable to dry-run traffic;
differences between simulated outcomes and sandbox/live test outcomes;
frequency with which review discovered a policy gap.

If reviewers accept nearly every proposal without inspecting it, the interface may be encouraging rubber-stamping. If proposals are repeatedly rejected for the same missing context, improve the data path or workflow instead of increasing the model’s autonomy. If a dry-run result never affects a release decision, the team may be collecting traces without conducting meaningful evaluation.

Keep a threshold for ending the pilot, but define it by evidence rather than one aggregate acceptance percentage. A narrow reversible label might require consistent agreement across ordinary cases; a financial action needs stronger assurance, tested idempotency, a compensation path, and authorization controls in addition to proposal quality.

Avoid an optimistic simulator

A simulator that returns success for every valid-looking call can make an agent appear safer than it is. It proves that the planner can produce a request, not that the real integration will accept it or that the workflow handles failure.

Model realistic failure classes in the simulator:

stale resource version;
permission revoked after planning;
duplicate request with the same idempotency key;
downstream validation rejection;
timeout after an action may have committed;
partial completion across two systems;
rate limit or temporary dependency outage.

Use recorded fixtures or controlled fault injection to exercise those paths. The simulator should return unknown when it cannot know the outcome, rather than pretending an external system would certainly succeed. If you need to test provider-specific behavior, use the provider’s sandbox or a dedicated integration environment with test data.

For multi-step workflows, simulate state transitions, not just tool responses. If a first step creates a draft and the next step sends it, the simulated draft should exist for the second step to inspect. If the send is configured to fail, the test should verify that the workflow preserves the draft and does not report completion. This catches orchestration bugs before a live side effect is enabled.

The simulator itself is test code with security implications. Make it impossible to fall through to production clients, and review its configuration when new write tools are added. A fake adapter that shares credentials with the live adapter is a warning sign.

Make the preview legible to the person reviewing it

A dry-run result is only useful if the reviewer can understand the proposed change. Show the target record, current values, proposed values, side effects, assumptions, and any policy checks that passed or failed. Use a diff for field updates. For a message or document, show the exact text that would be sent or saved.

Label the state unambiguously: “Preview only—no changes made.” If a live operation is available after review, use a separate confirmation action and show whether the proposal expires. Do not make a green checkmark mean both “the plan validated” and “the operation completed”; those are different facts.

Preserve the original input and the dry-run result if the reviewer rejects it, subject to the product’s retention policy. A rejection reason can improve the evaluation set and reveal missing context. Let reviewers correct the proposal and record whether the model or the policy was wrong, rather than treating every rejection as one generic thumbs-down.

For batch previews, summarize totals but keep each item inspectable. A user should be able to remove a single risky action from the batch before approving the rest. Keyboard navigation and accessible labels matter here: a reviewer who cannot distinguish the simulated state from the live state can approve the wrong thing even when the backend is perfectly guarded.

Log the reviewer’s final decision against the exact proposal version. Otherwise the audit record may describe a different action. Keep approval tied to the displayed diff. Invalidate it when inputs change.

10. Decide what dry-run mode is allowed to prove

Dry run can provide evidence about:

the model’s proposed tool and parameters;
prompt and policy behavior under recorded inputs;
whether deterministic validation accepts the plan;
how a human review interface will present the proposed change;
how the workflow handles simulated partial failures.

It cannot prove:

that a real provider will accept the request;
that permissions or resource state will remain unchanged until execution;
that a downstream API will be available;
that a model’s next output will be identical;
that the user approved a specific action;
that a simulated compensation will succeed in production.

Use a sandbox or controlled live test for integration behavior, and use a confirmation step for user authority. Keep the evidence boundaries clear in release reviews. “Dry run showed the agent would refund this order” is not “the refund is safe to automate.”

A rollout path that uses dry run well

Implement read-only behavior and collect representative cases.
Add structured proposals and a dry-run adapter for every write tool.
Compare proposals with human outcomes and investigate disagreements.
Test the dry-run guarantee with instrumented clients and failure injection.
Enable human-approved execution for a narrow action and cohort.
Revalidate permissions and current state at execution time.
Consider bounded automation only for low-risk, reversible actions with clear limits.
Keep a separate kill switch and record whether each operation was planned, simulated, approved, or executed.

The deciding principle is simple: dry run should make the proposed behavior visible while making the side effect impossible. If the system cannot prove both halves, the mode is a label, not a safety control.

Review the mode boundary whenever a new tool or background worker is added.