The first multi-tenant AI feature I saw go wrong didn't leak data. It leaked patience.
One customer ran a month-end reconciliation that fanned out into forty thousand classification calls over about ninety minutes. Nothing in the system was per-tenant, so those calls consumed the shared provider quota, the shared worker pool, and the shared retry budget. Every other tenant's assistant got slower, then started returning timeouts, then started returning apologies. The incident graph for the other four hundred customers looked exactly like an outage, and the root cause was one customer using the product as designed.
The second failure mode is worse and quieter: a retrieval query that forgot a WHERE tenant_id = $1, and a support assistant that confidently summarized another company's policy document. Nobody noticed for three weeks because the assistant's answer was plausible and the two customers happened to have similar policies.
Both failures come from the same root cause. Teams treat multi-tenancy as an authorization concern — check the tenant on the way in, done — when AI features have four independent resources that must be partitioned: context, spend, throughput, and credentials. They fail separately, they need separate mechanisms, and only one of them is really about permissions.
Why AI features break the usual tenant isolation model
In a CRUD app, tenant isolation is mostly a property of queries. Every table has a tenant column, every query filters on it, and a test suite checks that a user from tenant A can't read tenant B's invoice. The data flow is request → row → response.
An AI feature inserts a runtime assembly step in the middle. Before the model sees anything, your application:
- resolves the user and tenant,
- retrieves documents, which may be filtered by permissions and tenant,
- loads conversation history or memory, which is often cached under a key you chose months ago,
- renders a prompt that may include tenant-specific configuration,
- calls a provider under a shared credential,
- writes results into a cache, a trace, and a usage ledger.
Each of those steps has its own isolation failure. Worse, several of them are shared by design for performance reasons. Prompt caching rewards identical prefixes. Response caching rewards identical inputs. Connection pools and worker pools are shared because per-tenant pools are wasteful at small scale. Every efficiency mechanism is a potential crossing point.
🚨 Production warning: the dangerous bugs in multi-tenant AI are not "user A sees user B's data." They're "tenant A's content ends up in a shared prefix that tenant B's request reads from cache," and "tenant A's traffic consumes tenant B's quota." Neither shows up in an authorization test.
Separation one: context assembly
Context is everything you put in front of the model: instructions, retrieved chunks, memory, tool results, and examples. Every one of those is a tenant boundary.
Retrieval. The filter must be part of the query, not a post-processing step. Post-filtering is the classic mistake, and it fails in two ways: it wastes retrieval slots on documents you'll discard (so the tenant gets worse results than they should), and it tempts a later refactor into "just return the top-k" when someone is optimizing latency.
If you're on PostgreSQL with row-level security, push the tenant into the session and let the database enforce it. If you're on a vector store, the tenant filter needs to be a first-class part of the index design, not an afterthought in the query builder.
-- Assumes: documents(tenant_id, id, content, embedding), RLS enabled on tenant_id
SET LOCAL app.tenant_id = 't_8812';
SELECT id, content, 1 - (embedding <=> $1) AS score
FROM documents
WHERE deleted_at IS NULL
ORDER BY embedding <=> $1
LIMIT 8;
The SET LOCAL pattern matters: it's scoped to the transaction, so a pooled connection can't carry a tenant from one request to the next. If you set it without LOCAL, you've created a cross-tenant leak that only appears under connection reuse — which is to say, in production and never in tests.
Conversation history and memory. The cache key is the vulnerability. A key of conversation:{conversation_id} is fine if conversation IDs are globally unique and unguessable. A key of memory:{user_id} is not, if user IDs are only unique within a tenant, or if a user belongs to two tenants and their memory should not mix. Make the key include the tenant explicitly, even when you think it's redundant:
conv:{tenant_id}:{conversation_id}
mem:{tenant_id}:{user_id}:{scope}
Redundant keys cost a few bytes and eliminate an entire bug class. They also make cache flushes per tenant possible, which you'll want the first time a customer asks you to purge their data.
Few-shot examples and tenant configuration. Tenant-specific settings — brand voice, product names, custom policies — must go in a section of the prompt that is clearly tenant-scoped, and must never be interpolated into the shared instructions above the cache breakpoint. Which brings us to the subtle one.
The prompt cache is a shared channel, and it has rules
Prompt caching is one of the few features that changes both cost and latency by an order of magnitude, so everyone enables it. It's also the least understood tenant boundary in most systems.
The mechanics: the provider hashes a prefix of your request and reuses the computed state if the hash matches a recent entry. On Anthropic's API, caching is prefix-based across tools, system, and messages in that order, with a 5-minute default lifetime and an optional 1-hour lifetime at a higher write cost. Reads are priced at a fraction of base input — 0.1x on most models, lower on some — which is why a 90% hit rate feels like a pricing bug in your favor.
Now the tenant question. Two requests with different first tokens get different hashes and share nothing. Two requests with the same first ten thousand tokens share everything up to the breakpoint. So:
- A shared system prompt with no tenant data is safe and valuable. Cache it.
- A system prompt that begins with "You are the assistant for Acme Corp" is tenant-specific. It will never share a cache entry with another tenant, which is correct but means you get no cross-tenant benefit and should not budget for one.
- A prompt that puts the tenant name after a large shared block still shares the shared block. That's the efficient design: shared instructions first, tenant-specific content after the breakpoint.
The failure mode isn't cache poisoning in the security sense — providers key on the full prefix hash, so a different tenant produces a different hash. The failure mode is budgeting and observation: a team assumes an 85% cache hit rate across the fleet, sizes their cost model accordingly, and then discovers that per-tenant prefixes drop them to 40%, because each tenant has a distinct policy block near the front.
There's a second, sharper rule. Never put anything user-specific in the cached prefix, including a user's name, their account tier, or a timestamp. It doesn't leak, but it silently destroys the hit rate, and you'll spend a week blaming the provider. Anthropic's docs call this out directly: a breakpoint on a block that changes every request produces a write every time and never a read.
Response-level caching is a different animal and needs its own tenant key. If you cache model outputs by a hash of the prompt, include the tenant in the hash input even when the prompt text is identical. Two tenants asking the same question can legitimately get different answers because their entitlements differ, and an answer generated under tenant A's permissions must never be served to tenant B.
Separation two: cost attribution
Most teams can tell you their monthly AI spend within about 20%. Almost none can tell you which customer caused it. That's a problem for pricing, for support ("why is my bill up?"), and for detecting abuse.
The unit that matters is cost per successful task, per tenant. Not cost per token, and not cost per request.
Attribution needs a ledger written at request time, not reconstructed from invoices:
CREATE TABLE ai_usage_events (
id bigserial PRIMARY KEY,
occurred_at timestamptz NOT NULL DEFAULT now(),
tenant_id text NOT NULL,
workspace_id text,
feature text NOT NULL,
model text NOT NULL,
prompt_version text NOT NULL,
input_tokens integer NOT NULL,
cache_read_tokens integer NOT NULL DEFAULT 0,
cache_write_tokens integer NOT NULL DEFAULT 0,
output_tokens integer NOT NULL,
tool_calls integer NOT NULL DEFAULT 0,
latency_ms integer NOT NULL,
outcome text NOT NULL, -- success | error | escalated | rejected
unit_cost_usd numeric(12,6) NOT NULL
);
CREATE INDEX ON ai_usage_events (tenant_id, occurred_at DESC);
Four design notes that make this table worth having.
Store unit_cost_usd, not just tokens. Prices change, and contracts differ. Recording the computed cost at the time of the event makes historical reporting stable and lets you re-price a subset when you renegotiate.
Keep the token buckets separate. Cached reads, cache writes, and ordinary input are billed at different rates and, as noted earlier, different providers report them with different field semantics. Collapsing them into one input_tokens column makes it impossible to explain a cost drop after a caching change.
Record the outcome. A ledger that counts failed requests as usage will tell you a tenant is expensive when they're actually just unlucky. Attribution should support both "billed usage" and "value delivered," and those are different aggregations of the same rows.
Attribute shared work deliberately. If you run a batch job that re-embeds a shared corpus, or a cache warm-up that benefits all tenants, you need a policy: charge nobody, charge everybody proportionally, or charge the tenant who triggered it. Pick one and write it down. The default — dumping it into an "unallocated" bucket — means nobody ever looks at it.
With that table, the interesting reports become one query away: spend by tenant and feature for the month, cost per successful task, the top ten tenants by spend growth week over week, and the share of spend that went to retries and errors. That last one is usually the most surprising number in the building.
Separation three: throughput and admission control
Per-tenant rate limits are where most implementations stop, and where most of them are wrong.
Limit tokens, not requests. A request limit treats a 200-token classification and a 60,000-token document summary as equal. Ten requests per minute sounds generous until a tenant's integration sends ten long-context requests and consumes the entire organization's per-minute token allowance. Providers publish limits in both requests-per-minute and tokens-per-minute for exactly this reason; mirror the stricter of the two.
Reserve before you call. You cannot enforce a token budget after the fact. By the time you know the usage, the money is spent. Admission control has to reserve an estimate up front — input tokens measured by your tokenizer plus the max_output_tokens you're willing to allow — and settle against actual usage when the response arrives. The difference between reserved and actual is what you release back to the bucket.
A token bucket in Redis, using server time to avoid clock skew across workers:
-- KEYS[1] = ai_bucket:{tenant_id}
-- ARGV[1] = capacity (tokens), ARGV[2] = refill per second, ARGV[3] = tokens requested
local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill = tonumber(ARGV[2])
local want = tonumber(ARGV[3])
local t = redis.call('TIME')
local now = tonumber(t[1]) * 1000 + math.floor(tonumber(t[2]) / 1000)
local state = redis.call('HMGET', key, 'tokens', 'ts')
local tokens = tonumber(state[1])
local ts = tonumber(state[2])
if tokens == nil then tokens = capacity; ts = now end
local elapsed = math.max(0, now - ts) / 1000.0
tokens = math.min(capacity, tokens + elapsed * refill)
local allowed = 0
if tokens >= want then
tokens = tokens - want
allowed = 1
end
redis.call('HSET', key, 'tokens', tokens, 'ts', now)
redis.call('PEXPIRE', key, math.ceil(capacity / refill * 1000) + 1000)
return {allowed, math.floor(tokens)}
Call it before the provider call with want = input_tokens + max_output_tokens, then after the response, credit back max_output_tokens - output_tokens. Two details: the PEXPIRE keeps idle tenants from accumulating keys forever, and returning an integer avoids Lua's float truncation surprises in older Redis versions.
Separate concurrency from rate. A tenant can be well under their token budget and still saturate your worker pool with thirty parallel long-running agent runs. A per-tenant concurrency cap — semaphore, not rate limiter — is a different control and you need both. The rate limiter protects the provider; the concurrency cap protects you.
Have a global ceiling. Per-tenant limits that are individually reasonable can sum to far more than your provider contract allows. If five hundred tenants each get 200k tokens per minute and your org limit is 4 million, the arithmetic doesn't work, and the failure will be a provider 429 storm that hits everyone at once. Admission control needs a global bucket too, and when it's exhausted, the decision about who gets served should be a policy, not a race.
Fairness: the part rate limits don't solve
Token buckets give each tenant a ceiling. They don't give each tenant a share. Under contention, a tenant with a burst allowance can still monopolize the queue, and a tenant with a small allowance can be starved by a steady stream of large ones.
If your workload goes through a queue, the scheduler matters more than the limiter. Plain FIFO is the default in every queue library and it's the wrong choice for multi-tenant AI work: one tenant's forty-thousand-item batch fills the queue and every interactive request from every other tenant waits behind it.
The fix is a fair queue — weighted fair queueing or deficit round robin over per-tenant sub-queues. Each tenant gets a weight derived from their plan, the scheduler takes one item at a time from each eligible sub-queue in rotation, and a tenant that's idle doesn't accumulate credit it can spend later as a burst.
weights: enterprise=8, pro=3, free=1
ready: [ent_A(41 items), pro_B(2), free_C(1)]
schedule: A,A,A,A,A,A,A,A,B,B,B,C,A,A,A...
Two properties matter. A tenant with one item doesn't wait behind a tenant with ten thousand. And weights are relative, so adding a tenant doesn't require recomputing anyone else's absolute limits.
Interactive work usually needs a second axis: priority class, not just tenant weight. A user waiting on a chat response and a nightly re-classification job should not compete on the same queue even if they belong to the same tenant. Two priority classes, each internally fair across tenants, is a workable starting point.
Budget caps, and what happens when you hit one
Rate limits handle the minute. Budget caps handle the month. They fail differently and need different UX.
A soft cap — a percentage threshold — should notify someone before it's hit, not after. The notification needs to go to a person who can act: the workspace owner for a self-serve plan, the account manager for an enterprise contract.
A hard cap needs a defined behavior, and "return an error" is usually the worst option. Better degradations, roughly in order of preference:
- Route to a cheaper model that's adequate for the task, and label the output as such if quality differs.
- Queue non-urgent work until the next budget period, with a clear status in the UI.
- Disable the specific expensive feature while keeping the rest of the product working.
- Refuse with an actionable message and a path to raise the limit.
What you should never do is silently keep spending. The first invoice after an uncapped month is a trust incident, and it's the kind that ends contracts.
💡 Practical note: cap at the workspace level, not only the tenant level. Enterprise tenants usually have multiple teams, and a single team running an experiment shouldn't consume the whole company's allowance without anyone noticing.
Separation four: credentials and tools
The fourth partition is the one that gets skipped because it looks like ordinary security work. If your AI feature calls tools on behalf of a tenant — reading their CRM, writing to their ticketing system, querying their warehouse — every tool call must execute under a credential scoped to that tenant, with the tenant's permissions, not a service account with god rights.
The dangerous shortcut is a shared integration credential plus a tenant_id parameter that the model supplies. That's authorization decided by generated text. A prompt injection, a malformed argument, or a model that confuses two similar IDs turns into a cross-tenant read, and your audit log will show a legitimate service account doing it.
The pattern that holds: the model selects a tool and provides arguments that describe what it wants. The application resolves whose credentials to use from the authenticated session, and the tool executes with those. Tenant identity never comes from the model.
This is also where per-tenant rate limits belong at the tool layer, not just the model layer. A tool that calls a customer's own API is subject to that customer's limits, and hammering it because your global limiter allowed it is a support ticket you caused.
Testing isolation requires two tenants and a canary
Authorization tests check that user A can't read tenant B's data. Isolation tests need to check something different: that tenant A's request never influences tenant B's response.
A practical suite:
- Context canary. Seed tenant B with a document containing a unique nonsense token. Run a hundred varied queries as tenant A. Assert the token never appears in any response, any trace attribute, or any cache entry keyed for A.
- Cache key audit. Assert every cache write in the AI path has a tenant component in its key. This is a static check on a key-construction function, not a runtime test, and it catches the class of bug that runtime tests miss.
- Quota accounting. Send a known workload as tenant A and assert tenant B's bucket and ledger are unchanged.
- Credential probe. For each tool, attempt a call with a forged tenant argument and assert the application ignores it and uses the session's tenant.
- Fairness smoke test. Enqueue a large batch for tenant A alongside interactive requests for B and C, and assert p95 latency for B and C stays within the interactive SLO.
That last test is the one teams never write, and it's the one that would have caught the month-end incident.
When one tenant outgrows the shared pool
Eventually a customer's usage is large enough that sharing stops being efficient. The instinct is to give them "dedicated capacity," which sounds simple and is not.
There are four levels of separation, and each buys a different guarantee:
Shared everything. One provider credential, one worker pool, one vector index. Cheapest, and fine until a tenant's traffic becomes a meaningful fraction of the total. The guarantee is best-effort.
Logical isolation with dedicated budgets. Same infrastructure, but the tenant gets its own token bucket, concurrency semaphore, queue weight, and cache namespace. This is the right answer for most large customers, because it addresses the failure that actually hurts — one tenant degrading another — without duplicating infrastructure. It does not protect against a provider-level outage.
Dedicated provider capacity. A separate provider organization, deployment, or contract for a specific tenant, so their rate limits and their incidents are their own. This is expensive and operationally annoying: two sets of credentials, two sets of model versions, two sets of quota dashboards, and a migration path when the tenant leaves. It's justified when a contract requires it, or when the tenant's volume genuinely exceeds what a shared org limit can absorb.
Dedicated data plane. Separate indexes, separate storage, sometimes separate regions. This is a residency or regulatory decision more than a performance one, and it should be priced accordingly.
The trade-off people underestimate is version skew. Dedicated capacity means the tenant can end up on a different model snapshot or a different prompt version than everyone else, and now every quality comparison and every incident review has two baselines. If you go dedicated, version the tenant's configuration explicitly and include it in your evaluation runs, or you'll be debugging a difference that has nothing to do with the code.
Observability has to be tenant-scoped, and that's a cardinality problem
You can't operate any of the above without seeing it per tenant, and the naive implementation will melt your metrics backend.
Putting tenant_id on every span attribute and every metric label produces cardinality equal to tenants × features × models × versions × endpoints. At a few thousand tenants, that's millions of series, most of them sparse, and your observability bill will grow faster than your inference bill.
The split that works: traces carry tenant identity (they're already per-request and sampled), while metrics aggregate into bounded dimensions. For metrics, keep tenant-level series only for the handful of signals that drive alarms — tokens consumed, error rate, queue age, and spend — and aggregate everything else. If you need per-tenant dashboards for customers, build them from the usage ledger rather than from the metrics system.
Then alert on the ratios, not the absolutes. "Tenant X used more tokens than yesterday" is noise; "tenant X's share of total tokens exceeded 25% for ten minutes" is the month-end incident, detected. "Tenant X got a 429" is expected under load; "tenant X's 429 rate is above 20% while the global rate is under 1%" means your fairness policy is misconfigured.
A decision table for the four separations
| Dimension | Default at small scale | What changes at 50+ tenants | Primary failure |
|---|---|---|---|
| Context | Shared vector index with mandatory tenant filter | Per-tenant collections or RLS with tested policies | Retrieval returns another tenant's docs |
| Cache | Shared system prompt, tenant content after breakpoint | Per-tenant prefix budgets, hit-rate monitoring | Hit rate collapses; costs rise silently |
| Spend | Monthly invoice divided by usage | Per-request ledger with unit cost and outcome | Can't attribute cost; caps surprise customers |
| Throughput | One global bucket | Per-tenant token buckets + concurrency + fair queue | One tenant degrades everyone |
| Credentials | One integration service account | Per-tenant delegated tokens, no model-supplied identity | Model-driven cross-tenant access |
The pattern across all five rows is the same: the small-scale default works because contention and volume are low, and it fails not when you add tenants but when one tenant behaves differently from the others. A single large customer, a single batch job, a single integration with a retry storm.
Build the partition before you need it, and build it in the place where it can't be forgotten: the query, the key, the bucket, the credential resolution. Not in a code review checklist, and not in a prompt instruction telling the model not to mix tenants. The model isn't the enforcement point. Your infrastructure is.