Picture a support ticket with two agents working it. One is a triage agent that classifies incoming messages and updates priority. The other is a resolution agent that drafts a reply and, when the customer agrees, issues a refund.
The customer agrees at 14:02:11. The resolution agent starts a multi-step sequence: read the ticket, check the order, call the refund tool, then write a summary back to the ticket. That sequence takes eleven seconds, because two of those steps are model calls.
At 14:02:14, a new customer message arrives. The triage agent reads the ticket, decides the issue is now "billing dispute" rather than "refund request," and writes back the whole ticket object it read at 14:02:13 — a snapshot that predates the resolution agent's summary. At 14:02:22, the resolution agent writes its summary.
The refund went through. The summary says it went through. The ticket's category says "billing dispute," the status reverted to "open," and the assignee is now whoever the triage agent thought it should be. Nobody did anything wrong in code review. Both agents used the ordinary read-modify-write pattern that every web application uses, and it worked fine when the only writer was a human clicking Save.
Why agents turn a solved problem back into an unsolved one
Concurrent edits are old territory. Databases have transactions, HTTP has conditional requests, and every ORM ships optimistic locking. The problem is that all of those mechanisms were designed around interactions measured in milliseconds to a few seconds, with a human in the loop who can see a conflict banner and decide what to do.
Agents break three of those assumptions at once.
Think time is enormous and variable. A single agent turn can be 800 ms for a classification call or 45 seconds for a reasoning-heavy tool loop. The window between reading a record and writing it back is not a few milliseconds; it's the length of a model call, sometimes several. Every one of those seconds is a window for another writer.
The read and the write are in different tool calls. A human filling a form has the record in front of them and submits once. An agent typically calls get_ticket, reasons, then calls update_ticket. Between those calls, the model may make three other tool calls, retry one, and re-read the record — or not re-read it at all, because the earlier read is still in its context and looks perfectly valid.
Retries are automatic and invisible. Provider timeouts, 429s, and tool failures all trigger retries somewhere in the stack. A retried update_ticket is not a read-modify-write anymore; it's a blind write with stale arguments. And a retried issue_refund is a second refund.
The result is a class of bug that's hard to reproduce, hard to see in logs, and completely invisible in a single-agent test suite.
🚨 Production warning: if two instances of the same agent can run for the same record — a retry, a duplicate queue delivery, a user clicking twice — you already have two agents. Concurrency isn't only a multi-agent problem.
The failure taxonomy, in order of how often they bite
1. Lost update
Two writers read version 7, both compute a new state, both write. The second write overwrites the first, and the first agent's work disappears with no error anywhere.
This is the most common failure and the easiest to fix. It's also the one teams assume can't happen because "the agent always re-reads before writing." It doesn't, and even when it does, the re-read is followed by more reasoning time.
2. Whole-object overwrite
A subtler variant: the agent read the record early in the conversation, built a complete replacement object, and wrote all fields back. Even if the record hasn't changed since, the agent's copy is missing fields that were added after it read — a new escalation_reason, a sla_deadline set by a background job, a customer_sentiment field another agent populated.
Whole-object writes are the default in most tool schemas because they're easy to describe to a model. They're also the reason one agent's unrelated field edit vanishes.
3. Write skew
Two agents each enforce an invariant by reading and then writing a different row. Example: a policy says an order can have at most one active refund. Agent A reads "no active refunds for order 991," decides to create one. Agent B does the same thing at the same moment. Both inserts succeed, and now there are two refunds, each individually valid at the time it was created.
This one is nasty because no single row was updated concurrently. Row-level locking doesn't help; you need a constraint, a lock on a shared parent, or serializable isolation.
4. Acting on a stale snapshot
The agent reads the invoice at 10:00, the customer pays at 10:01 via a different path, and at 10:03 the agent sends a payment reminder. No write conflict occurred — the agent's write was to a different field, or to a notification table — but the action was wrong because the premise was stale.
Optimistic locking doesn't catch this, because there's no version check on "send email." The only defenses are precondition checks close to the side effect and, for high-stakes actions, a re-read immediately before acting.
5. Duplicate side effects
The agent calls issue_refund, the call times out at the HTTP layer after the server processed it, the agent retries, and the customer is refunded twice. This is the classic idempotency problem, and agents make it worse because they retry at multiple layers: the tool wrapper, the model's own retry decision, and the orchestration framework.
6. Interleaved multi-step sequences
Agent A's plan is: check inventory → reserve item → charge card. Agent B's plan is: check inventory → reserve item → charge card. Both check, both reserve what they think is the last unit, both charge. Each step was individually correct. The sequence wasn't atomic and nothing enforced the constraint across it.
Agents make this pattern common because plans are explicit and multi-step by nature. A human would notice "wait, this is the last one"; the agent has no such moment.
7. Semantic conflict with a clean merge
Two agents edit different fields and the merge succeeds technically. Agent A sets shipping_address to the address in the customer's latest email. Agent B sets it to the address on the account, because it decided the email was a phishing attempt. Both writes apply to different fields — one to address_line_1, one to address_source — and the record now contradicts itself.
Field-level merging solves lost updates and creates this instead. There's no mechanical fix; you need either a single owner for the record's semantic state or a review step.
The latency math that kills pessimistic locking
The obvious fix for all of this is a lock: acquire before read, release after write. And it fails on the arithmetic.
If an agent turn takes eight seconds and you hold a row lock for the whole turn, that record supports 7.5 edits per minute per record, and every other writer blocks — including the UI that a human is looking at. Worse, the lock is held across network calls to a model provider, which is exactly the pattern every database guide tells you to avoid. A provider hiccup becomes a lock pileup; a lock pileup becomes a connection pool exhaustion; that becomes an outage in a system that had nothing to do with the original problem.
PostgreSQL's SELECT ... FOR UPDATE holds until transaction end, and advisory locks are session-scoped, which interacts badly with connection poolers in transaction mode. Neither is designed for "held during a 20-second HTTP call."
So the design has to be layered: cheap, optimistic concurrency for ordinary field edits, and short-lived leases only for operations that genuinely must be exclusive. The mistake is applying one mechanism to everything.
| Mechanism | Hold time | Survives holder crash | Works across services | Fits an 8s agent turn |
|---|---|---|---|---|
SELECT ... FOR UPDATE |
Transaction lifetime | Yes, on rollback | No | No |
| Advisory lock | Session lifetime | Only via connection close | No | No |
| Redis lock without fencing | TTL | No, unsafe | Yes | Short holds only |
| Lease + fencing token | TTL, checked on write | Yes | Yes | Yes |
| Version + conditional update | None | N/A | Yes | Yes |
The row that matters is the last one. Optimistic concurrency holds nothing, so it has no throughput cost and no crash semantics to reason about. That's why it should carry the majority of your write traffic, with leases reserved for the small set of operations where two simultaneous executions would be a disaster.
Optimistic concurrency is the default, and it's cheap
Every mutable record gets a version integer (or a updated_at timestamp with sufficient resolution, though an integer is less error-prone). Reads return it. Writes require it.
UPDATE tickets
SET status = $1,
assignee_id = $2,
version = version + 1,
updated_at = now()
WHERE id = $3
AND version = $4
RETURNING version;
Zero rows returned means someone else wrote first. That's not an error to swallow — it's a signal that the agent's plan was based on a state that no longer exists, and the correct response is to re-read, re-reason, and retry with new information. Bounded, with a limit; an agent that retries a conflict forever is its own incident.
The important part is what the tool exposes. If your update_ticket tool takes a version and returns a conflict as a structured, recoverable error, the model can adapt: "the ticket changed while I was working; here's the current state; adjust." If the tool silently overwrites, the model never learns there was a conflict.
from dataclasses import dataclass
@dataclass
class Conflict:
code = "stale_version"
retryable = True
def update_ticket(conn, ticket_id: str, expected_version: int, patch: dict) -> dict:
sets = ", ".join(f"{k} = %s" for k in patch)
values = list(patch.values())
row = conn.execute(
f"UPDATE tickets SET {sets}, version = version + 1, updated_at = now() "
f"WHERE id = %s AND version = %s RETURNING version, status, assignee_id",
values + [ticket_id, expected_version],
).fetchone()
if row is None:
current = conn.execute(
"SELECT version, status, assignee_id, category FROM tickets WHERE id = %s",
(ticket_id,),
).fetchone()
return {"ok": False, "error": Conflict.__dict__, "current": dict(current)}
return {"ok": True, "version": row[0]}
Two details worth copying: the conflict response includes the current state, so the agent doesn't need a second call to recover, and the update is a field-level patch rather than a whole-object replacement. Both reduce the chance that the next attempt loses someone else's work.
The f"UPDATE ... {sets}" construction is safe only because patch keys come from a fixed allowlist of column names, never from model output directly. Build the allowlist explicitly; don't let a tool schema turn into SQL injection with extra steps.
Leases and fencing tokens for the operations that must be exclusive
Some operations can't be merged or retried. Issuing a refund, sending a legally binding notice, deleting a record, transferring ownership. For those, you need mutual exclusion, and you need it to survive a crashed holder.
A lease is a lock with an expiry. A fencing token is a monotonically increasing number issued with the lease, which the protected resource checks on write. The token is what makes the lease safe: a holder that paused for 60 seconds (GC, network stall, model provider timeout) still has a lease object, but its token is stale, and the write is rejected.
CREATE TABLE record_leases (
record_id text PRIMARY KEY,
holder text NOT NULL,
fencing_token bigint NOT NULL,
expires_at timestamptz NOT NULL
);
Acquire, allowing takeover only when the existing lease has expired:
INSERT INTO record_leases (record_id, holder, fencing_token, expires_at)
VALUES ($1, $2, nextval('lease_fence'), now() + interval '45 seconds')
ON CONFLICT (record_id) DO UPDATE
SET holder = EXCLUDED.holder,
fencing_token = EXCLUDED.fencing_token,
expires_at = EXCLUDED.expires_at
WHERE record_leases.expires_at < now()
RETURNING fencing_token;
And the guarded write checks that the token is still current:
UPDATE invoices
SET status = 'paid', version = version + 1
WHERE id = $1
AND version = $2
AND EXISTS (
SELECT 1 FROM record_leases l
WHERE l.record_id = $1
AND l.fencing_token = $3
AND l.expires_at > now()
);
Set the lease duration from the operation's p99, not its median. If the refund sequence takes 6 seconds at p50 and 22 seconds at p99, a 45-second lease is reasonable; a 10-second lease produces spurious takeovers under normal load, and spurious takeovers mean two agents doing the exclusive thing at once — exactly what the lease was for.
The cost is throughput. A 45-second lease on a record permits roughly one exclusive operation per 45 seconds on that record. That's usually fine, because exclusive operations are rare. It is not fine if you apply leases to everyday field edits, which is why the two mechanisms have to coexist.
Idempotency keys belong on every tool, not just payments
The duplicate-refund problem has a known solution: the caller supplies a key, the server records it, and a repeated call with the same key returns the original result instead of doing the work again.
Agents need this on far more than payments. Any tool whose effect is visible outside the transaction needs a key: sending an email, posting a comment, creating a calendar event, opening a ticket in another system, incrementing a counter. The key should be derived from the decision, not from the attempt, so a retry produces the same key.
idempotency_key = sha256(f"{tenant_id}:{agent_run_id}:{action}:{target_id}")
Deriving from agent_run_id rather than from a random UUID per attempt is what makes retries safe. Deriving from the decision rather than the payload means a model that reformats its arguments on retry doesn't accidentally create a second effect.
Store the key with the response, not just with the fact that it was used. A retry should get the same answer the first attempt got, including the identifier of the thing it created.
Append-only intent logs for the cases that need negotiation
There's a category of conflict that no locking scheme resolves: two agents with incompatible goals, both acting reasonably. One wants to close the ticket as resolved; the other has just discovered a second issue and wants to reopen it.
For these, the useful pattern is to stop treating the record as the unit of truth and treat the sequence of intentions as the truth. Each agent appends an intent event — {actor, action, target, rationale_code, at} — and a projection service folds the events into current state using explicit conflict rules: last-intent-wins per field, or "reopening beats closing," or "escalate to a human when two intents within 60 seconds disagree."
That's more machinery than most features need. But it has one property that's hard to get otherwise: the conflict is preserved as data rather than resolved by whoever wrote last. When someone asks why the ticket ended up closed after a reopen, the log tells the story, and you can change the resolution rule later and replay.
The tool schema is where most of this is decided
It's tempting to treat concurrency as a database problem and stop there. But the shape of your write tool determines which failures are even possible, and it's much cheaper to fix the schema than to add coordination later.
Three rules.
Patches, not documents. A tool called update_ticket that accepts a complete ticket object forces every write to be a whole-object overwrite, which means every concurrent edit to an unrelated field is at risk. A tool that accepts only the fields the agent intends to change makes unrelated-field loss structurally impossible. If your model provider supports strict JSON schemas for tool arguments, use them, and make every field optional except the identifier and version.
Distinguish "not provided" from "set to null." This is the detail that breaks patch semantics in practice. In JSON, {"assignee": null} usually means "clear the assignee," while {"assignee": absent} means "leave it alone." A model that includes every field with null for the ones it didn't think about will clear half the record. Either omit absent fields in the schema and instruct the model accordingly, or use an explicit clear_fields array alongside the patch. Don't rely on the model remembering the difference.
Return the version you wrote. The write response should include the new version and the fields that changed. That gives the agent an accurate picture for its next step without a second read, and it gives your logs a record of what actually landed.
The same logic applies to read tools. If get_ticket returns the version, the model has what it needs to make a conditional write. If it doesn't, you've made optimistic concurrency impossible to implement from the agent side, no matter what your database supports.
💡 Practical note: version numbers in tool responses are also a cheap diagnostic. If your traces show agents repeatedly hitting
stale_version, the record has more concurrent writers than the workflow assumes, and the fix might be workflow design rather than retry tuning.
Conflicts your database can't see
Some of the worst cases involve two systems of record and a database that's perfectly consistent with itself.
An agent updates a CRM record and calls the billing provider. The billing call succeeds, the CRM write fails on a version conflict, and the agent retries the whole sequence — except the retry starts from the fresh CRM state and issues a second billing call. Locally, every transaction was correct. Globally, the customer was charged twice.
The same shape appears with any external effect: sending email, creating a shipping label, posting to a partner API. Optimistic locking protects your row, not the world.
Two practical mitigations. First, order the steps so external effects happen after the local state transition that records intent, not before. Write "refund requested, idempotency key X" locally, commit, then call the provider, then write "refund confirmed." A crash between steps leaves a reconcilable record instead of an orphaned charge. Second, run a reconciliation job that compares external state to local state on a schedule and flags divergence. Agents will produce divergence; the question is whether you find it in an hour or in a customer complaint.
What the user should see when it happens
Concurrency bugs in agent features usually surface as confusing product behavior rather than errors: a summary that contradicts the status, two notifications about the same event, a field that reverts.
The UI has to be honest about three states that most designs collapse into one.
In progress. The agent is mid-sequence on a record someone else is also touching. Show it. A small "updating…" indicator that names the agent, plus a disabled save button, prevents the human from creating a third concurrent writer at exactly the wrong moment.
Partially applied. The local write succeeded, the external call failed. This is not an error state and should not be presented as one. It's "refund recorded, confirmation pending," with a retry path and a timestamp. Users tolerate pending; they don't tolerate silent inconsistency.
Conflicted. The agent's write lost. If the agent can recover automatically, it should, and the user should see the final result. If it can't, the conflict needs a human decision with both versions visible: what the agent intended, what the current state is, and which fields differ. A generic "something went wrong" here is a support ticket.
The pattern I'd default to for anything with real consequences is the draft: the agent writes to a draft layer, the UI shows a diff against current state, and a human or a deterministic policy commits it. That doesn't eliminate concurrency, but it moves the conflict resolution to a moment where a person can see both sides — which is where it belongs for refunds, contract changes, and anything a customer will read.
What to apply to which record
| Mechanism | Protects against | Cost | Use for |
|---|---|---|---|
| Version column + conditional update | Lost update, whole-object overwrite | Near zero | Every mutable record |
| Field-level patch | Unrelated field loss | Small | All agent write tools |
| Precondition check at side effect | Stale-snapshot actions | One extra read | Notifications, status changes |
| Idempotency key | Duplicate side effects | One table, one TTL | Every external effect |
| Lease + fencing token | Concurrent exclusive operations | Throughput on one record | Money, deletion, ownership |
| Append-only intent log | Unresolvable intent conflicts | Significant | Multi-agent workflows on shared objects |
| Serializable isolation | Write skew | Retry handling everywhere | Invariants across rows |
Test with two agents, on purpose
Single-agent tests will never find these bugs. You need a harness that runs two agent instances against the same record with controlled interleaving.
The cheapest version doesn't need real models. Replace the model with a scripted sequence of tool calls, and inject a delay between the read and the write. Then assert three things: the final state is one of the acceptable outcomes, no write was silently lost, and every side effect happened exactly once.
def test_concurrent_status_update(client, ticket):
a = scripted_agent(client, steps=[
("get_ticket", {"id": ticket.id}),
("sleep", {"ms": 500}),
("update_ticket", {"id": ticket.id, "status": "resolved"}),
])
b = scripted_agent(client, steps=[
("get_ticket", {"id": ticket.id}),
("sleep", {"ms": 200}),
("update_ticket", {"id": ticket.id, "category": "billing"}),
])
run_concurrently(a, b)
final = fetch_ticket(ticket.id)
assert final.status == "resolved"
assert final.category == "billing"
Run this before you ship any write-capable tool, and run it again whenever the tool schema changes. The test is fast, deterministic, and it catches the exact class of bug that takes four hours to reconstruct from production logs.
Then add the adversarial cases: a lease holder that stalls past expiry, a tool that times out after committing, a retry with reformatted arguments. Those are the ones that produce incidents rather than test failures.
The broader point is that an agent writing to a record is not a special case of a human writing to a record — it's a client with unusually long think time, unreliable retries, and no visual awareness of concurrent edits. Design the write path for that client, and the human path gets safer as a side effect.