The model started emitting refund confirmations for orders that had never been paid. The on-call engineer did the reasonable thing: reverted the deploy. Eleven minutes later the revert finished rolling out, and the confirmations stopped. Eleven minutes is a long time when the feature handles money.
Here's the part that stung. The bad behavior wasn't caused by the deploy. A provider had silently updated a model alias behind the same version string, and the prompt that had worked for three weeks now interpreted "confirm the refund" as "tell the customer the refund is complete." Reverting the code changed nothing. The team had a deploy button where they needed a kill switch, and the deploy button controlled the wrong variable.
This is the central confusion: a deploy is a change to code; a kill switch is a change to behavior. They have different latency budgets, different failure modes, different authorization, and different testing requirements. Conflating them means your fastest available response to an incident is measured in minutes and depends on a build pipeline that may itself be degraded.
Deploy, flag, circuit breaker, kill switch
These four mechanisms get used interchangeably in design docs and they are not the same thing. Getting the distinctions right is most of the design work.
| Mechanism | Trigger | Scope | Latency | Reversible by |
|---|---|---|---|---|
| Deploy / rollback | Human, via pipeline | Whole artifact | Minutes to tens of minutes | Engineer with deploy rights |
| Feature flag | Human or experiment system | Targeted cohort | Seconds | Product or engineer |
| Circuit breaker | Automatic, on error rate or latency | One dependency | Milliseconds | Code, on recovery |
| Kill switch | Human, under pressure | Deliberately broad | Seconds, ideally sub-second | Incident commander |
A feature flag is not a kill switch, even though it can be wired to behave like one. Flags are built for targeting and gradual rollout; they're evaluated in the request path, they have a control plane, and they accumulate. A kill switch is built for one thing: making a feature stop doing work, immediately, with a decision that is easy to make at 3 a.m.
A circuit breaker is also not a kill switch. It reacts to observed failure and closes automatically when the failure clears. That's the right behavior for a flaky dependency and the wrong behavior for "this feature is producing legally risky output." You don't want automatic recovery from a policy decision.
The distinction that matters operationally: a kill switch must be trustworthy when everything else is broken. If the switch depends on the same database, the same config service, or the same cloud region as the feature, it isn't a kill switch. It's a second thing that goes down with the first.
Decide what "off" means before you need it
"Turn it off" is ambiguous, and ambiguity at 3 a.m. costs minutes. Write down the semantics per feature.
Hard off. No model calls, no tool calls, no writes. The endpoint returns a static response or a 503 with a friendly message. This is the default for anything touching money, permissions, or external side effects.
Degrade. Skip the AI path and serve a deterministic fallback — cached answers, a search results page, a form. The feature still exists; it just stops using the model. This is usually better for user experience and worse for clarity, because "degraded" can look like "working" to everyone except the person who set it.
Drain. Stop accepting new work, let in-flight work finish, then stop. Appropriate for batch jobs and queue consumers. Dangerous for interactive features where in-flight requests can take 30 seconds.
Scope-limited off. Disable for one tenant, one model, one tool, or one region. This is where most real incidents land: one customer's data triggers a bad path, or one provider region is degraded.
Pick one primary semantic per feature and document the others as explicit variants. A switch with three meanings and no labels will be set to the wrong one.
The switch has to be readable in the request path without a network call
The naive implementation reads a boolean from Redis or the database on every request. That's one extra dependency in the hot path and a new failure mode: when Redis is slow, your emergency stop gets slow too.
The pattern that holds up is a control plane plus a local cache with a bounded staleness window:
- The authoritative state lives in a control-plane table with an audit trail.
- Each service polls or subscribes, keeping a local copy in memory.
- The request path reads memory only. No I/O, no network, no lock contention.
- A stale copy is used for at most
max_stale_ms. Beyond that, the service falls back to the compiled-in default, which is chosen per feature:offfor spend and side effects,onfor read-only paths where availability matters more.
import time
import threading
from dataclasses import dataclass
@dataclass(frozen=True)
class SwitchState:
enabled: bool
updated_at: float
reason: str
class KillSwitch:
"""Reads from memory. Refreshed out-of-band by a poller."""
def __init__(self, name: str, default_enabled: bool, max_stale_s: float = 30.0):
self.name = name
self._default = default_enabled
self._max_stale_s = max_stale_s
self._state = SwitchState(default_enabled, time.monotonic(), "compiled default")
self._lock = threading.Lock()
def update(self, enabled: bool, reason: str) -> None:
with self._lock:
self._state = SwitchState(enabled, time.monotonic(), reason)
def enabled(self) -> bool:
with self._lock:
state = self._state
if time.monotonic() - state.updated_at > self._max_stale_s:
return self._default
return state.enabled
The max_stale_s bound is the important line. Without it, a service that loses contact with the control plane keeps the last value it saw forever — which is fine until the last value was "on" and the control plane is telling everyone else "off." With it, the worst case is bounded and explicit.
The default matters just as much. For a feature that spends money or writes data, default_enabled=False means an isolated instance with no control-plane contact fails safe. For a read-only assistant, True may be correct so a partition doesn't take down the product. There's no universal answer; there is a required decision.
⚠️ Gotcha: A switch read once at process start is not a switch. It's a config value. If flipping it requires a restart, you've rebuilt the deploy button with extra steps.
Where in the request you check it
Placement determines what the switch can actually prevent. Checking at the top of the handler stops the model call. It does not stop a queued job that was enqueued two seconds before the flip, or a tool call that was already in flight.
Check at every boundary that costs money or changes state:
- Before the model call. Cheapest point; prevents spend.
- Before each tool call. Prevents side effects from a multi-step agent that started before the flip.
- Before enqueueing work. Prevents a queue of doomed jobs from building up.
- Inside the worker, at the start of each job. Catches work enqueued before the flip.
- Before the final write or send. The last line of defense for irreversible actions.
A multi-step agent can run for a minute. Without checks 2 and 5, disabling the feature stops new conversations but lets existing ones keep issuing refunds. That's usually not what the incident commander meant.
There's a cost: an extra in-memory read per step is negligible, but the code gets noisier. Wrap it in a guard that raises a typed exception so the control flow is obvious:
class FeatureDisabled(RuntimeError):
pass
def require_enabled(switch: KillSwitch, step: str) -> None:
if not switch.enabled():
raise FeatureDisabled(f"{switch.name} disabled at {step}")
Then let the top-level handler map FeatureDisabled to a clean 503 with a retry hint, or to the degraded path if one exists. Workers should treat it as a non-retryable outcome and acknowledge the message, not requeue it — otherwise the kill switch converts into a hot loop.
Failing closed without failing rude
The user-facing behavior of a kill switch is part of its design. A raw 500 is unacceptable for a feature that might be off for an hour; a fake success is worse.
Three levels, in order of preference:
Serve the deterministic fallback. If the assistant is off, show the help-center search. If the classifier is off, route to the default queue. Users get a working product and never learn the model was involved.
Say it's unavailable, plainly. "The assistant is temporarily unavailable. Your message wasn't sent." Honest, short, no blame, no fake reassurance. Avoid "our AI is thinking extra hard" style copy, which ages badly and looks deceptive in a screenshot.
Refuse with a path forward. For irreversible operations, tell the user what to do instead: "Refund confirmations are paused. Contact support and we'll process it manually." The fallback should not be a dead end.
What you must not do is return a cached or templated answer that looks personalized. If the switch exists because the model was producing wrong account-specific claims, serving a plausible-looking cached claim is the same failure with extra steps.
Authorization and audit: the switch is a loaded weapon
A kill switch that any engineer can flip is a liability in the other direction: one mistaken toggle can take down a revenue feature during a launch. Treat flipping as a privileged action with a low-friction but recorded path.
Reasonable defaults:
- On is easy, off is easier. Disabling should require no approval. Enabling after an incident should require a second person or an incident reference.
- Reason is mandatory. A free-text field with a ticket link. The audit row is worthless without it.
- Every flip is logged with identity, time, scope, previous state, and new state. Ship it to the same audit store as your other privileged actions.
- Broad scope requires broader rights. Disabling one tenant is an on-call action; disabling globally is an incident-commander action.
- No silent auto-enable. If you build automatic re-enable, require a cool-down and a notification. Automatic recovery from a manual stop is how incidents recur at 4 a.m.
A useful audit query is "who turned this off in the last 90 days, and what happened next?" If nobody can answer that, the switch is a placebo.
Test the switch the way you test backups
An untested kill switch has an unknown activation time and an unknown blast radius. Both are discoverable, and both are worth discovering before an incident.
Quarterly game day, 15 minutes:
- Pick a non-critical feature.
- Start a timer.
- Have the on-call flip the switch using only the runbook.
- Confirm the effect in production metrics — model calls per minute should drop to zero within one staleness window.
- Flip it back. Confirm recovery.
- Record the elapsed time and any confusion.
The number you get is your real time-to-disable. If it's over 60 seconds, the runbook or the tooling is the problem, not the people. Teams that do this usually discover that the switch exists in three places with different names, that the dashboard doesn't show the switch state, or that the person with permission is on a plane.
Two automated checks are also worth having:
Coverage check in CI. A static rule that every registered KillSwitch is referenced in at least one guarded entry point, and that new AI endpoints declare a switch. Without this, switches drift into irrelevance as the codebase grows.
State check in monitoring. Export killswitch_enabled{name=...} as a gauge and alert on unexpected state changes. A switch silently flipped by a config typo is an outage with no deploy to blame.
💡 Practical note: Put the kill switch state on the same dashboard as the feature's error rate and spend. During an incident, the first question is "is it off?" and the answer should take one glance, not one query.
Why AI features need this more than most features
Plenty of software ships without a kill switch. A reporting page can be broken for an hour. AI features are different in five specific ways, and each one raises the cost of a slow response.
Behavior changes without a deploy. A prompt is fetched from a database. A model alias moves to a new snapshot. A tool server changes its response shape. A retrieval index gets rebuilt with different chunking. None of these produce a new artifact in your repo, so none of them are caught by "what changed in the last deploy?"
Output is probabilistic. A conventional bug reproduces. A model regression may hit 2 percent of requests and look like noise in the error rate. By the time it's visible in aggregate, it has affected thousands of users, and the fix isn't a code revert.
Cost can run away in minutes. An agent that loops, a retry storm, or a prompt that grows with context can multiply spend by 50x before an alert fires. The switch is the only control that stops the meter immediately.
Side effects are real. An agent with write access can issue refunds, send emails, change records, and call third-party APIs. The window between "this is wrong" and "this is expensive to undo" is short.
The failure can be a safety or legal event. Discriminatory output, a leaked document, a fabricated medical claim. In those cases, "we'll have a fix in the next sprint" isn't a response.
The common thread: AI systems have more inputs that change outside your deploy pipeline than conventional software, and their failures are more likely to be continuous rather than binary. That combination is exactly what a runtime kill switch is for.
Scoping: global, per-model, per-tenant, per-tool
A single global boolean is better than nothing and worse than almost any scoped design, because the first real incident is usually narrow.
Per feature. The base unit. support_assistant, refund_agent, document_classifier. Each has an owner, a runbook, and a default.
Per tenant. One customer's data triggers a bad path — a language the model handles poorly, a document type it misreads, a contract with unusual terms. Disabling for that tenant is proportionate and keeps everyone else working. This requires the tenant ID to be available at the guard, which it usually is but sometimes isn't in async workers; check that before you need it.
Per model or provider. When a provider is degraded or a model version misbehaves, route to a fallback or disable the model-dependent paths. This is closer to a routing decision than a kill switch, but it uses the same mechanism and should share the same audit trail.
Per tool. The most underrated scope. An agent with six tools can keep five and lose one. refund_tool off, lookup_tool on, with the prompt updated to say refunds are unavailable. This turns a whole-feature disable into a partial degradation, which is often enough.
Per region. Data residency and provider outages are regional. If your architecture already routes by region, the switch should be scoped the same way, or you'll disable a healthy region to stop a broken one.
Scope resolution should be deterministic and documented: the most specific matching rule wins, and the effective state is computed by the control plane and pushed to services. Don't make the request path evaluate a rule tree; make it read one resolved boolean per scope it cares about.
The runbook is part of the feature
A kill switch without a runbook is a button with no instructions. The runbook needs to fit on one screen.
FEATURE: refund_confirmation_agent
IMPACT WHEN OFF: refunds are not confirmed; support queue handles manually
DISABLE
1. Open the switch console, search "refund_confirmation_agent"
2. Set scope = global, state = off, reason = incident link
3. Verify: dashboard "refund agent calls/min" drops to 0 within 30s
4. Post in #incidents: what changed, when, expected user impact
VERIFY THE EFFECT
- SELECT count(*) FROM agent_tool_calls
WHERE tool_name = 'issue_refund' AND created_at > now() - interval '5 min';
- Check queue depth: refund_confirmations should drain to 0
RE-ENABLE
- Requires incident commander approval
- Re-enable at 5% traffic for 30 minutes before global
- Watch: refund error rate, customer complaints, tool error rate
Two details make the difference between a runbook that works and one that gets ignored. The verification step must be a concrete query or dashboard, not "check that it's working." And re-enable must be a ramp, not a flip, because the condition that caused the incident may still be present.
Also write down what the switch does not stop. If refunds can still be issued through the admin UI, saying so prevents a false sense of containment.
Backstops for when the switch itself fails
Defense in depth means assuming the switch is unavailable, unauthorized, or simply didn't cover the path that broke.
Provider spend limits. Set a hard monthly or daily cap in the provider console. It's crude — it turns a spend problem into an availability problem — but it bounds the worst case.
API key revocation. A separate, faster lever than the application switch. Revoking the key stops all traffic from that credential in seconds, at the cost of stopping everything else that uses it. Keep AI credentials separate from other provider credentials so revocation is surgical.
Network egress rules. For high-risk deployments, block egress to the model provider at the network layer. This is the only control that works if the application is compromised, and the only one that a compromised service can't override.
Queue pause. Independent of the application, pause the consumer. Useful when the failure is in processing rather than generation.
Model alias pinning. If you can, call a pinned model version rather than a moving alias. This doesn't replace a kill switch, but it removes one class of "nothing changed and everything changed" incident.
Each backstop has a different blast radius. The point is that at least one of them should be operable by someone who doesn't have deploy rights, and at least one should work when the application is unhealthy.
Measure time-to-disable, not time-to-deploy
Most teams know their deploy duration to the minute and have no idea how long it takes to stop a feature. That's backwards for incident response.
Track three numbers per AI feature, and review them quarterly:
- Time to detect. From the first bad output to the first alert or complaint. Usually the longest segment, and the one observability work improves.
- Time to decide. From detection to someone with authority saying "turn it off." This is a process and culture metric; it shrinks when disabling is cheap and blameless.
- Time to effect. From the decision to the last unwanted model call. This is pure engineering: staleness window plus in-flight work plus queue drain.
If time to effect is over a minute, fix the mechanism. If time to decide is over ten minutes, fix the escalation path — usually by pre-authorizing on-call to disable without approval, and by making it obvious that disabling is always the safer default during ambiguity.
The number to publish is the sum, and the target should be stated in the incident review template. "We turned it off in 40 seconds" is a much better sentence than "we reverted in 12 minutes and it didn't help."
The anti-patterns that keep showing up
The switch that requires a deploy. Implemented as an environment variable baked into the image. Fastest response is a full pipeline.
The switch behind the feature. The kill switch endpoint is part of the same service it disables. When the service is down or saturated, the switch is unreachable.
The per-request switch. A database read on every request. Adds latency, adds a failure mode, and during a database incident the switch is the first thing to break.
The switch with no default. When the control plane is unreachable, behavior is undefined. Usually it means the last value persists, which is the opposite of fail-safe if the last value was "on."
The one-way switch. Easy to turn off, impossible to turn on without a deploy. The incident ends, the feature stays dark for a week, and someone builds a workaround.
The switch nobody can find. Named disable_v2_tmp, undocumented, owned by someone who left. This is common enough that naming and ownership should be part of the switch registry.
The switch that hides the failure. Turning it off makes the dashboard green while the underlying model behavior is still broken. Track "feature disabled" as a first-class state in your SLOs, not as healthy.
A minimum viable kill switch, in order
If you're building this from scratch, do it in this order and stop when you've covered your risk.
- Inventory the AI entry points. Every endpoint, queue consumer, cron job, and tool executor. You can't guard what you haven't listed.
- Define the off semantics per entry point. Hard off, degrade, or drain. Write it in the runbook.
- Add one global switch per feature family, with a compiled default of off for anything that spends or writes.
- Read it from memory with a bounded staleness window and a poller that is independent of the feature's dependencies.
- Check it at every money-or-state boundary, not just at the top of the handler.
- Log flips with identity, reason, and scope; require a second person to re-enable.
- Put the state on the main dashboard and alert on unexpected changes.
- Run the game day and publish the time-to-disable number.
- Add the CI coverage check so new endpoints can't ship unguarded.
- Add one backstop you can operate without the application — key revocation or an egress rule.
Steps 1 through 4 are a day of work for a small team. The rest is operational discipline, and it's where most of the value is, because a switch that exists but can't be found at 3 a.m. is indistinguishable from no switch.
The decision that matters
Ask one question in the design review: if this feature started causing harm in the next ten minutes, what is the fastest action we could take, who can take it, and what does it stop?
If the answer is "roll back the deploy in about fifteen minutes, if the pipeline is healthy, and it won't help if the cause is upstream," you don't have a kill switch. You have a deploy button and a hope.
If the answer is "the on-call flips one toggle, it takes effect within thirty seconds, it stops model calls and tool calls and queue intake, and the fallback is a help-center search," then you've separated behavior control from code control. That separation is the whole point. Deploys are for changing what the software is. Kill switches are for deciding what it's allowed to do right now.