A safety filter that blocks every risky-looking sentence can look excellent in a dashboard. It can also prevent a patient from asking for crisis resources, a security engineer from analyzing malware, or a customer from quoting the threat they received. “Flagged” is a signal. It is not, by itself, a product decision.
Guardrails are often described as a wall between a model and the user. That picture encourages one threshold and one outcome: pass or block. Real systems need more choices. Some inputs should be rejected. Some need a clarification. Some generated claims need to be checked against trusted data. Some actions require a human. And some content that looks dangerous in isolation is entirely legitimate in context.
The goal is not to catch every conceivable bad output at any cost. It is to reduce harmful outcomes while preserving legitimate use, with a clear reason for each intervention and a way to learn from mistakes. That requires layers, policy, and measurement—not just a moderation call in front of the model.
Start with the harm, not the filter
Before adding a guardrail, define the outcome you are trying to prevent. “Unsafe content” is too broad to implement. For a customer-support assistant, the relevant harms might be disclosing another tenant’s information, promising an unauthorized refund, giving abusive replies, or exposing secrets in a trace. Those failures need different controls.
Write a small threat model around the feature:
- Input: What can users submit, including images, documents, code, and quoted text?
- Context: Which retrieved documents, profile fields, or tool outputs influence the answer?
- Output: What could the model say that would cause harm or mislead the user?
- Action: Can the model trigger a write, payment, message, permission change, or external call?
- Audience: Who can be affected, and how severe is the consequence?
That framing reveals that some “AI safety” problems are ordinary application-security problems. Tenant isolation belongs in authorization checks, not in a prompt asking the model not to reveal other customers’ records. Payment limits belong in code, not in a classifier that scores the model’s intention. A content filter can help with harmful text, but it cannot replace access control or business rules.
The NIST AI Risk Management Framework is a voluntary framework rather than a turnkey guardrail specification, but its lifecycle view is useful: identify context and risk, measure, manage, and continue monitoring after deployment. A guardrail decision should be traceable to a concrete risk in the feature, not to a generic “AI is risky” checkbox. (NIST AI RMF)
Use more than two outcomes
Binary allow/block decisions create avoidable false positives. A more useful policy has at least five outcomes:
| Outcome | When it fits | Product response |
|---|---|---|
| Allow | No relevant risk signal | Continue normally |
| Transform | The request is valid but needs constrained handling | Redact, normalize, or limit scope |
| Clarify | The input is ambiguous or missing a key fact | Ask a targeted question |
| Review | Consequence is significant or confidence is insufficient | Queue for a human decision |
| Block | The requested action is disallowed or clearly harmful | Refuse briefly and offer a safe alternative where possible |
For example, an HR assistant asked to “summarize this employee complaint” should not block merely because the text contains harassment. It can summarize the report while preserving the distinction between a quoted allegation and the assistant’s own language. The same content in a request to generate targeted abuse may need to be blocked. Context changes the product decision.
The policy should say which cases are allowed, transformed, reviewed, or blocked. A model’s category score can inform that policy, but it should not silently become the policy.
Put checks at the layer that can enforce them
An effective guardrail stack is not a sequence of prompts. It is a set of controls placed where the application can enforce them.
1. Validate input shape and authority
Reject oversized payloads, malformed files, unexpected MIME types, unsupported languages if the product cannot handle them, and fields the caller is not allowed to set. Bind user identity and tenant scope from authenticated server-side context; do not accept them from model-generated arguments.
For sensitive actions, authenticate and authorize before any model call that could access or change data. A model should not be able to expand the caller’s permissions by producing a convincing explanation.
2. Treat user-provided content as untrusted data
Prompt injection often arrives inside a document that the system is supposed to summarize, an email it is supposed to classify, or a webpage it is supposed to search. A content moderation model may classify harmful language, but it does not reliably determine whether every instruction in a document should be trusted.
Separate instructions from data. Keep retrieved text in a clearly delimited data section. Do not grant tools more authority because a retrieved page says “ignore previous instructions.” Enforce tool permissions in code, validate arguments, and require approval for sensitive writes. A prompt can help the model interpret the boundary; the application must make the boundary real.
3. Constrain outputs before they reach downstream systems
Use typed schemas for structured outputs. Validate enumerations, numeric ranges, identifiers, and required fields. If a generated SQL query or email body will be used downstream, validate it according to that domain before executing or sending it.
For a free-form answer, check claims against available source data where possible. If the answer includes a delivery date, compare it with the order API result. If it contains a price, use the price service as the source of truth. A general-purpose “hallucination detector” is not a substitute for verifying facts that your application can check directly.
4. Moderate content as one signal
Moderation systems can detect categories of potentially harmful content in text and, depending on the service and model, images. They are useful as one input to policy routing. They do not know your entire product context, and their scores can shift as the underlying model changes.
OpenAI’s current moderation guide describes omni-moderation-latest for text and image inputs, and supports either standalone classification or moderation results alongside a generated response. It explicitly recommends treating moderation scores as signals for your application’s policy rather than automatic blocking decisions. The guide also warns that custom policies based on category_scores may need recalibration as the endpoint’s underlying model changes. (OpenAI, Moderation)
The same guide notes an important limitation: the moderation model does not classify audio, and its child-safety guidance says not to treat the endpoint as a CSAM detection system. Know the exact modalities and categories your chosen control covers. “We moderate all user input” is not an accurate statement if audio or other content types bypass that check.
5. Gate side effects separately from language
An answer can be acceptable while the action is not. “I found the correct account” does not authorize changing its billing plan. A model’s refusal or confidence score should not control access to a payment API.
Before a tool call, enforce:
- the authenticated actor and tenant;
- the allowed action and resource;
- field-level constraints and amount limits;
- idempotency for retries;
- current resource state;
- approval requirements;
- audit logging of the proposal and decision.
This is where deterministic policy earns its keep. Even if the model is manipulated by hostile input, code should still reject an unauthorized write.
Avoid guardrail theater
Several controls sound reassuring but do not enforce the property people assume they enforce.
“The system prompt says not to do it.” That instruction shapes generation; it does not authenticate a user, restrict a database query, or prevent an unsafe tool call. Keep the instruction, but put the permission check in the tool implementation.
“The model checks its own answer.” A second pass can catch some omissions, but the same model may share the first pass’s blind spot. Treat self-review as a quality step, not as an independent safety boundary.
“We moderate the prompt.” User input is only one place risk enters. A retrieved document can contain hostile instructions, a tool can return corrupted data, and a generated answer can make a claim not present in the input. Decide whether checks belong before retrieval, after generation, and before each side effect.
“The output is valid JSON.” Syntax does not imply safe meaning. A valid object can contain another tenant’s ID, a negative refund amount, or a prohibited action. Parse it, validate it against the authenticated context, and enforce business constraints.
The architecture should make it possible to bypass the model without bypassing the guardrail. For example, the payment service should reject an unauthorized refund whether the request came from an agent, an admin UI, a queue worker, or a test script. A safety rule that exists only in one prompt is a rule that other callers do not have.
This separation also improves incident response. If a harmful output appears, you can ask whether the content classifier missed it, the policy router chose the wrong action, or the application allowed a tool call. “The guardrail failed” is not yet a useful diagnosis.
Tune thresholds for the consequence, not for symmetry
Every classifier has false positives and false negatives. A false positive blocks a legitimate user. A false negative allows a harmful case through. The costs are rarely equal, and they vary by category.
For example:
- A harassment filter for a public comment composer may favor preventing targeted abuse, while still providing a review path for reports and quotations.
- A fraud signal that only queues a transaction for review can tolerate more false positives than a signal that automatically closes an account.
- A content filter for a medical education product should distinguish a request for general information from an emergency request; blocking both is not “safer.”
- A detector for secret leakage should favor blocking a confirmed credential, but a generic string pattern may also catch harmless examples in documentation.
Measure precision and recall separately for each important category. Precision asks how many flagged cases were actually problematic. Recall asks how many problematic cases were detected. A single accuracy score can look high while the filter misses the rare harm or blocks a large share of legitimate inputs.
Thresholds should be calibrated on examples that reflect the product’s real context: ordinary benign inputs, difficult but legitimate cases, policy violations, quoted material, and adversarial attempts. Review the confusion matrix by language, modality, user segment, and risk. A threshold that works for short English text may behave differently on a long multilingual document.
Do not choose a threshold by maximizing accuracy alone. Imagine a classifier that sees 1,000 benign requests and 10 requests that truly need intervention. A rule that allows everything scores well on overall accuracy, yet misses every case the guardrail exists to catch. Conversely, a rule that blocks all 1,010 requests catches every risky example and destroys the product. Report precision, recall, and the base rate, then make the trade-off explicit for each route.
For a review route, the cost of a false positive is reviewer time and delay. For a hard block, it is a denied task and potentially a lost customer. For an irreversible action, a false negative can trigger a financial or privacy incident. The same score threshold should not necessarily govern all three outcomes. Set thresholds from a cost matrix and measure the human workload they create.
Also separate “flagged” from “must block.” A moderation result can include category flags and scores; a refusal response may itself discuss harmful material, so output moderation may flag content that is a safe refusal. Read the categories and context before applying a policy. If the moderation call returns an error instead of a result, handle that state explicitly rather than treating the missing score as zero.
Make interventions understandable and recoverable
A generic “I can’t help with that” frustrates users when the request is legitimate. A detailed explanation of internal policy may expose sensitive rules or invite adversarial probing. The user-facing response should be specific enough to be useful without revealing implementation details.
Prefer messages such as:
- “I can summarize the complaint, but I can’t identify the person who filed it. Remove names or ask an authorized reviewer to open the case.”
- “I couldn’t verify the invoice total from the file. Check the highlighted amount or upload a clearer copy.”
- “This action needs approval from a billing administrator. I’ve saved the draft and sent it to the review queue.”
These responses distinguish a policy block from missing information and from a system failure. They also preserve work where possible. If the user can correct a field, appeal a classification, or request human review, show that path clearly.
For every block or intervention, decide what happens to the input and output. Is the draft retained? Is it sent to a reviewer? Is it included in telemetry? Can the user retry after editing? These details affect both privacy and trust. Do not silently retain sensitive content because a guardrail flagged it.
Guardrail responses also need to work for people who rely on assistive technology, are using a mobile screen reader, or have a slow connection. Put the reason in text, not color alone; move keyboard focus to the intervention message; preserve the user’s original input; and make the next action discoverable. A modal that announces “blocked” but loses the text the user spent five minutes composing is a product failure even if the policy decision was correct.
Avoid accusatory language when the system is uncertain. “This request may need a review” is more accurate than “You violated policy” when a classifier has only a probabilistic signal. Provide a specific correction or appeal path when feasible. If users repeatedly need to appeal the same category, that is evidence about the threshold, policy definition, or missing context—not just an individual user problem.
Keep the intervention proportional. A suspicious phrase in a quoted support ticket may justify a review marker; it does not necessarily justify locking the account. Separate the consequence for the content from the consequence for the user, and require stronger evidence for punitive actions than for routing a draft to a reviewer.
Use a risk router instead of a universal threshold
A practical guardrail can route by category and action rather than returning a single boolean. The example below is intentionally application-level: it assumes moderation is already a normalized result from a provider, and request is validated server-side. It does not treat a model score as authorization.
from dataclasses import dataclass
from enum import StrEnum
class Decision(StrEnum):
ALLOW = "allow"
CLARIFY = "clarify"
REVIEW = "review"
BLOCK = "block"
@dataclass
class Request:
intent: str
has_write_action: bool
actor_can_write: bool
has_required_context: bool
@dataclass
class Moderation:
flagged: bool
categories: dict[str, bool]
def decide(request: Request, moderation: Moderation) -> Decision:
if not request.actor_can_write and request.has_write_action:
return Decision.BLOCK
if not request.has_required_context:
return Decision.CLARIFY
if moderation.categories.get("self-harm/intent", False):
return Decision.REVIEW
if moderation.categories.get("harassment/threatening", False):
return Decision.REVIEW
if moderation.flagged:
# A generic flag is a signal to inspect context, not an automatic deny.
return Decision.REVIEW
return Decision.ALLOW
This policy is only illustrative. A real product needs domain-specific rules, a defined response for each category, and a path for moderation errors. Some situations require immediate safety resources; others require a human reviewer; still others are false alarms that should pass. Do not use this example as a universal safety policy.
The important separation is visible: authorization is deterministic; missing context leads to clarification; category flags route to review; only a policy that understands the product decides to block.
Fail safely when the guardrail itself fails
Moderation services and policy services can time out. Decide what the application does before an outage occurs.
The answer depends on the operation. If a user is asking for a low-risk summary, the application might return a temporary error or offer a reduced feature. If the request would publish content publicly or execute a financial write, it may be appropriate to stop until the safety check succeeds. A blanket fail-open policy is unsafe; a blanket fail-closed policy can make the entire product unusable during an upstream incident.
Use category-specific failure behavior:
- for an irreversible write, block execution and preserve a pending draft;
- for a low-risk informational response, explain the temporary limitation or use a deterministic fallback;
- for an urgent user-safety path, provide the service’s established emergency response rather than waiting on a classifier;
- for a review queue, record the dependency failure and make the item visible to operators.
Apply timeouts, bounded retries, and circuit breakers to the guardrail dependency. If retries are used, make sure they do not duplicate the downstream action. Record whether a decision came from moderation, deterministic policy, a human, or a fallback so incidents can be reconstructed.
Evaluate good users as carefully as bad inputs
Security testing tends to focus on whether a malicious request gets through. Product quality also depends on whether legitimate users can still complete their work. Your eval corpus needs both sides.
For each policy category, include:
- clearly disallowed examples;
- benign examples that share vocabulary with disallowed content;
- quotation, reporting, and educational contexts;
- ambiguous examples that should clarify or review;
- multilingual and modality-specific cases the feature claims to support;
- repeated or paraphrased attacks;
- cases where a moderation flag is irrelevant to the requested action.
Measure block precision, block recall, review rate, user correction rate, appeal overturn rate, and completion rate for legitimate tasks. If you only measure blocked harmful content, a filter that blocks everyone can look perfect. If you only measure completion, a system that ignores its policy can look efficient.
OpenAI’s evaluation guidance recommends task-specific datasets that reflect real traffic and human calibration for automated graders. The same principle applies to guardrails: evaluate the actual application decision, not just whether a model emitted a category flag. (OpenAI, evaluation best practices)
Monitor for drift and policy changes
Guardrails are not “set and forget.” The user population changes, attack patterns change, and moderation models can be updated. A threshold that worked on last quarter’s data may create a spike in false positives after a model change.
Monitor:
- flag and block rates by category and input type;
- human review volume and queue age;
- appeal and overturn rates;
- policy errors and downstream incidents;
- latency and dependency failure rate;
- performance by language and user segment where lawful and appropriate;
- version of moderation model, prompt, and policy code.
Changes to thresholds or category mapping should go through a review process and a staged rollout. Sample both blocked and allowed cases; otherwise you only inspect the visible side of the boundary. Keep a rollback path for the policy configuration, not just the application deploy.
Treat raw prompts and moderation inputs as sensitive data. Store the minimum needed for investigation, redact credentials and personal data, restrict access to traces, and apply a retention period. A guardrail log that copies every dangerous input into a broad analytics system can create a new security problem while attempting to prevent another one.
Keep a versioned record of the policy threshold, moderation model identifier, and application code for each decision. During an incident, a label like “blocked by safety” is not enough to reproduce why a particular request was routed or why another similar one passed.
When the moderation provider changes its underlying model, rerun the same release set before relying on the new scores. Treat a score shift as a behavior change even if the endpoint name stayed constant, and stage the update where you can observe false-positive and false-negative patterns before broad rollout.
Keep a small rollback-ready sample of known benign and known disallowed cases for quick smoke checks after configuration changes.
For high-impact categories, require an owner to approve threshold changes and retain the previous configuration for immediate rollback. Review that rollback path during routine operational exercises.
The practical rule
Place every guardrail where it can actually enforce its promise:
- input limits and schemas at the request boundary;
- authorization at the data and tool boundary;
- moderation as a contextual signal;
- business rules in deterministic code;
- human review for consequential ambiguity;
- output checks before display or execution;
- monitoring and appeals after deployment.
Then test both failure directions. Does the system catch the harmful action? Does it still let a legitimate user complete a difficult but allowed task? A guardrail is successful only when it reduces the risk it was designed for without quietly turning ordinary use into a sequence of false alarms.
The strongest safety design is not the one with the most blocking rules. It is the one that knows when to allow, constrain, clarify, review, or refuse—and can explain which path it took without giving the model authority it should never have had.