A support assistant has been talking to the same customer for forty minutes. Somewhere around turn twelve the customer mentioned they're on the annual Pro plan, billed in euros, and that the invoice must go to a different address than the card. At turn thirty the assistant confidently recommends a feature that only exists on Enterprise, quotes a price in dollars, and asks for a billing address it already has.
Nothing crashed. No error was logged. The assistant was working exactly as designed: on every request it took the last three thousand tokens of history, dropped everything older, and sent the request. The dropped part contained the only three facts that mattered.
This is the quiet failure mode of conversational AI features. Everyone builds trimming, because context windows are finite and token spend is real. Almost nobody measures what trimming costs in correctness, because the cost doesn't show up as an exception. It shows up as a support ticket that says "the bot forgot what I told it."
What context actually costs, in numbers you can act on
Before choosing a trimming strategy, it helps to be precise about the thing you're trimming.
Every turn of a conversation, you resend the whole prompt: system instructions, tool definitions, the conversation so far, and the new user message. Input tokens are billed per request, so a ten-turn conversation doesn't cost ten times one prompt, it costs the sum of a growing series. If each turn adds roughly k tokens and the base prompt is b, the total input for n turns is:
total_input = n*b + k*n(n-1)/2
That quadratic term is the whole story. With b = 1,500 tokens of system prompt and tool schemas and k = 250 tokens per exchange, ten turns cost about 26,000 input tokens. Forty turns cost about 255,000. The conversation didn't get ten times more valuable, but it got ten times more expensive, and the growth accelerates.
Prompt caching changes the shape of this curve but doesn't remove it. Anthropic's API requires explicit cache_control breakpoints, exactly matching a prefix, with cache lifetimes measured in minutes (five minutes by default, one hour as an option). OpenAI applies automatic caching to prompts above a minimum length, also keyed on an exact prefix match. In both cases the discount applies only to the unchanged prefix, and only if your trim logic keeps that prefix byte-stable. A trimming routine that rewrites the system prompt per turn — injecting a "current time" line, or a fresh list of the user's projects — invalidates the cache on every request and quietly costs more than no cache at all.
So there are two costs, and they pull in opposite directions. Longer history costs more tokens. Aggressive trimming that mutates the front of the prompt costs cache hits. A workable design has to respect both.
🧠 The important part: context is not a budget you spend down. It's a state you have to maintain. The moment you treat it as a buffer to truncate, you've decided to lose information without deciding which information.
Why "keep the last N tokens" is the wrong default
Token-window truncation is popular because it's three lines of code and it never throws. It's also the strategy most likely to produce the failure above, for a structural reason: recency has almost nothing to do with importance in a task-oriented conversation.
Consider what actually lives in a long support conversation:
- Durable constraints stated once — plan tier, locale, currency, account ID, accessibility preferences, "don't email me, I'm traveling."
- Commitments the assistant made — "I'll escalate this and you'll hear back within 24 hours."
- Diagnostic facts discovered mid-conversation — the error code, the browser, the fact that clearing cookies didn't help.
- Tool results — often the bulkiest content and the least useful once interpreted.
- Small talk and acknowledgements — genuinely disposable.
Only the last category is safe to delete by age. Everything else is a candidate for deletion by recency, and every one of them causes a visible failure when dropped.
There's a second, subtler problem: truncation breaks pronoun and reference resolution. "Move it to the second one" is meaningless without the turn that established what "it" and "the second one" refer to. Cut in the middle of a reference chain and you don't lose a fact, you lose the ability to interpret the next message. The model doesn't know it's missing something; it guesses, fluently.
Separate durable state from the transcript
The single highest-leverage change is to stop treating the conversation as the system's memory. The conversation is an interface. Memory should be a structured object you control.
A conversation state record might look like this:
from dataclasses import dataclass, field
from typing import Any
@dataclass
class ConversationState:
# Facts the user asserted. Never trimmed; superseded explicitly.
facts: dict[str, Any] = field(default_factory=dict)
# What the assistant promised or committed to.
commitments: list[str] = field(default_factory=list)
# Things the user explicitly excluded or refused.
exclusions: list[str] = field(default_factory=list)
# Open questions the assistant still needs answered.
open_questions: list[str] = field(default_factory=list)
# Rolling summary of turns no longer sent verbatim.
summary: str = ""
# Index of the first turn still present in the raw transcript.
transcript_start: int = 0
Then the prompt is assembled from parts with different lifetimes:
- System instructions and tool schemas — static, cacheable, never rewritten per turn.
- Rendered state — small, structured, regenerated each turn (it's cheap and it's the part that must be fresh).
- Rolling summary — updated only when compaction runs.
- Recent transcript — the last few turns verbatim.
- The new user message.
The critical detail is that the render of the state goes after the cacheable prefix, not inside it. If your state block is injected into the system prompt and changes every turn, you've destroyed the prefix cache. Put it in its own message immediately before the recent transcript, or after it, and keep the ordering stable.
def build_messages(state: ConversationState, recent: list[dict], user_msg: str) -> list[dict]:
return [
{"role": "system", "content": SYSTEM_PROMPT}, # static, cached
{"role": "user", "content": render_state(state)}, # changes per turn
{"role": "assistant", "content": "Understood."},
*recent,
{"role": "user", "content": user_msg},
]
That trailing assistant acknowledgement looks like a hack, and it is one, but it's a useful one: it keeps the alternation valid and it gives the model a clean boundary between "facts I was given" and "things that were said."
The payoff is that dropping a turn no longer loses the facts in it, because facts were never only in the turn. They were extracted into state the moment they were asserted.
Extract facts at write time, not at trim time
Fact extraction is the step most teams skip, because it feels like an extra model call. It is — one small, cheap, structured call per turn, or per few turns. It's worth it for a simple reason: extraction happens while the evidence is present and unambiguous, whereas summarization-at-trim-time happens later, over a window that may already have lost the context needed to interpret it.
EXTRACT_PROMPT = """You maintain a state record for a support conversation.
Current state:
{state_json}
New exchange:
user: {user_msg}
assistant: {assistant_msg}
Return JSON with only the fields that changed:
{"facts": {...}, "commitments": [...], "exclusions": [...], "open_questions": [...]}
Rules:
- A fact is a durable attribute of the account or the user's intent, not conversational filler.
- If new information contradicts an existing fact, put the new value in "facts" and the old key in "exclusions" with a note.
- Do not invent values. Empty objects and lists are valid and common.
"""
Two rules matter more than the prompt wording. First, contradictions must be represented, not silently overwritten. If the user said euros and later says dollars, the state should record that the currency changed, and ideally when. Overwriting produces an assistant that's confidently wrong about which is current. Second, negations are first-class. "Don't call me, email only" is not a preference, it's a constraint, and it belongs in a list that's never trimmed and always rendered. Most trimming bugs I've seen in review are actually negation-loss bugs.
⚠️ Gotcha: extraction models love to promote passing remarks into facts. "I might upgrade next year" becomes
plan: enterprise. Store speculative statements asnoteswith a source turn ID, and give the answering model the raw note rather than a normalized fact. A normalized fact that's wrong is much harder to detect than a note that's vague.
Compact at semantic boundaries
When you do need to compress the transcript, choose boundaries by meaning rather than by token count.
A task-oriented conversation has natural seams: the customer's problem is resolved, the topic changes, a tool call sequence completes, the user says "okay, different question." Compacting at a seam produces a summary that hangs together. Compacting at token 3,000 produces a summary whose first sentence is the back half of a sentence about something else.
In practice, a simple heuristic gets most of the value: compact when the accumulated transcript exceeds a threshold and the last exchange contains no unresolved tool call and no question awaiting an answer. If either condition fails, wait a turn. Being slightly over budget for one request is cheaper than compacting mid-task.
def should_compact(turns: list[dict], tokens: int, limit: int) -> bool:
if tokens < limit:
return False
last = turns[-1] if turns else {}
if last.get("tool_calls"):
return False # don't cut a tool loop in half
if last.get("role") == "assistant" and last.get("awaiting_user"):
return False # the answer depends on what comes next
return True
The summary itself should be written for a reader who has to continue the conversation, not for a reader who wants to know what happened. That means it should preserve commitments verbatim, keep numbers exact, and keep the user's own wording for anything emotional. "Customer is frustrated that this is the third contact about the same charge" is more useful downstream than "customer expressed dissatisfaction."
Keep the tail verbatim, and keep it short
A rolling summary plus a short verbatim tail beats a long summary for one reason: the tail carries the style and shape of the conversation, which the summary can't. Tone, formality, the specific words the user used for their problem, the IDs that were just read out — all of that lives in the last few turns.
Three to six turns is usually enough. The tail's job is continuity, not recall.
TAIL_TURNS = 4
def compact(state: ConversationState, turns: list[dict]) -> tuple[ConversationState, list[dict]]:
if len(turns) <= TAIL_TURNS:
return state, turns
to_summarize = turns[:-TAIL_TURNS]
new_summary = summarize(state.summary, to_summarize)
state.summary = new_summary
state.transcript_start += len(to_summarize)
return state, turns[-TAIL_TURNS:]
And the summarizer prompt should be explicit about what must survive unchanged:
SUMMARIZE_PROMPT = """Merge the existing summary with these older turns.
Preserve exactly, without paraphrasing:
- dollar amounts, dates, IDs, error codes
- any promise the assistant made about timing or next steps
- any constraint the user stated (contact method, plan, locale, access needs)
Drop: greetings, acknowledgements, restated questions, and the contents of
tool results that were already interpreted.
Write in past tense, third person, under 200 words. Do not add interpretation
that is not supported by the turns.
"""
The "do not add interpretation" line is doing real work. Summarizers drift toward narrative — they explain why the user was upset, which is exactly the kind of content that's plausible and unverifiable, and which the next turn will treat as established fact.
Drop tool payloads, keep their conclusions
Tool results are consistently the largest and least reusable part of a long history. A single search that returns twelve documents can be 8,000 tokens; the model used 200 of them to answer. Keeping the raw result in history costs you the ability to keep three more real turns.
The rule that works: once a tool result has been consumed, replace it with a distilled artifact. Not a summary of the JSON — the specific fields that were referenced, plus a pointer to the full record if the model or a human needs to go back.
def distill_tool_result(tool_name: str, raw: dict) -> dict:
if tool_name == "search_orders":
return {
"tool": tool_name,
"count": len(raw["results"]),
"top": [
{"id": r["id"], "status": r["status"], "total": r["total"]}
for r in raw["results"][:5]
],
"note": "full results retrievable with get_order(id)",
}
return {"tool": tool_name, "raw_ref": raw.get("id"), "keys": list(raw)[:20]}
This is also an observability improvement. A transcript full of distilled results is readable by a human during an incident; a transcript full of raw search payloads is not.
🔍 Why this matters: distillation is where you decide what the system considers evidence. If you distill away the field that later turns out to matter, you've created a bug that only appears on the conversations where that field mattered — the hardest kind to reproduce.
Retrieval over history, for the cases that need it
Some conversations genuinely need recall of something said forty turns ago, verbatim. Summaries won't do it. The answer is not a bigger window; it's retrieval.
Embed each turn (or each coherent exchange) as it's stored, and when the current message references something older — an order ID, a previous ticket, "the thing we discussed about refunds" — search the history and inject the matching exchanges as a clearly labelled block:
Earlier in this conversation:
[turn 6] user: The refund was supposed to go to the card ending 4417.
[turn 7] assistant: Confirmed, refund initiated to card ending 4417 on 3 June.
Two design notes. Label retrieved history as history, not as current instruction — otherwise a retrieved user message can be read as a fresh command, which turns your own log into a prompt-injection vector. And retrieve by identifier, not just by semantic similarity: exact-match lookup on order IDs and ticket numbers is cheap and far more reliable than embeddings for the cases that matter most.
Measure the cost of trimming, not just the token bill
You cannot tune this without an evaluation set that includes long conversations. The single most useful artifact is a small set of real (or realistically constructed) conversations, each with a fact planted early and a question that requires it late.
Three metrics cover most of the ground:
| Metric | How to compute | What it catches |
|---|---|---|
| Fact retention | Replay the conversation; ask the late question; check the answer against the planted fact | Over-aggressive trimming, summary drift |
| Constraint adherence | Plant a negation early ("don't email"); check whether later turns violate it | Negation loss, state rendering bugs |
| Cost per resolved conversation | Sum input tokens across all turns, divide by conversations resolved | Trimming that's too conservative, cache misses |
Replaying a conversation turn by turn is the only honest way to measure this, because trimming decisions compound. A single-request eval tells you nothing about turn thirty.
def replay(conversation: list[dict], agent, checks: list[dict]) -> dict:
state = ConversationState()
turns: list[dict] = []
results = []
for event in conversation:
user_msg = event["user"]
reply, turns, state = agent.step(state, turns, user_msg)
for check in checks:
if check["at_turn"] == event["index"]:
results.append({
"check": check["name"],
"passed": check["predicate"](reply, state),
})
return {"results": results, "input_tokens": agent.total_input_tokens}
Run it with trimming disabled, then with each candidate strategy, and compare. The differences are usually larger than expected, and they're usually in the direction of "the cheap strategy is losing facts you didn't know you needed."
When the summary and the state disagree
Inevitably, they will. The summary says the customer is on Pro; the state says the plan changed to Business last week. The summary says the refund went out; the state's open_questions still lists "refund status unknown."
Two rules keep this from becoming a slow-motion correctness bug.
The structured state wins over the prose summary, always. The summary is a compression of the transcript, and the transcript is a record of what was said. The state is a record of what is currently true. When they conflict, the ordering in the prompt should make the hierarchy explicit — render the state under a heading like Current account facts (authoritative) and the summary under Earlier conversation (may be outdated). Models follow stated precedence surprisingly well when it's stated once, plainly, and consistently.
Conflicts should be logged, not resolved silently. A counter on summary/state disagreement is a leading indicator that your extraction prompt is missing a field, or that the summarizer is inventing. Both are cheap to fix early and expensive to discover from a customer complaint.
def state_summary_conflicts(state, summary):
"""Cheap heuristic: does the summary assert a plan/currency/status
that the state contradicts?"""
conflicts = []
for key in ("plan", "currency", "contact_method"):
if key in state.facts and key in summary:
stated = summary[key]
actual = state.facts[key]
if stated and str(stated).lower() != str(actual).lower():
conflicts.append((key, stated, actual))
return conflicts
A production version of this would use the same extraction model to read the summary back into a structured form and diff it against state. It costs a few hundred tokens per compaction and catches the class of bug where the summarizer gradually rewrites history.
The compaction prompt is a spec, so version it
There's a reason this section exists between the metrics and the strategy: the summarizer prompt is not configuration. It's the definition of what your system remembers, and it should be reviewed, versioned, and tested like any other code that changes behavior.
Concretely, that means the prompt lives in the repository next to the trimming logic, gets a version string that's stored alongside each generated summary, and has fixture conversations in the test suite. When someone edits the "preserve verbatim" list, the fixtures show what changed in the produced summaries:
def test_preserves_amounts_and_ids():
turns = [
{"role": "user", "content": "The charge was $184.20 on 3 June, order A-99120."},
{"role": "assistant", "content": "I see it. I'll refund it today."},
]
summary = summarize(previous="", turns=turns + filler(20))
assert "184.20" in summary
assert "A-99120" in summary
assert "refund" in summary.lower()
Storing summary_prompt_version with each summary also gives you a migration path. When you change the prompt, old summaries were produced by the old spec; you can either leave them (accepting mixed semantics) or re-summarize from the raw transcript on next compaction. For long-lived conversations, re-summarizing lazily is usually better than leaving two generations of summary logic in play.
The same argument applies to the extraction prompt. Facts extracted by v1 and v3 of a prompt aren't the same facts, and rendering them into the same state object without any marker makes debugging harder than it needs to be.
🚨 Production warning: if you ever change what the summarizer considers a commitment, the assistant's promises will start disappearing from long conversations. That looks like a model quality regression and gets escalated to the wrong team. Version-stamp the summary so the investigation has somewhere to start.
A trimming strategy you can actually ship
The combination that holds up across support assistants, coding copilots, and internal tool agents:
- Static prefix for system prompt and tool schemas. No per-turn content. Cache it.
- Structured state extracted every turn (or every other turn for low-stakes conversations), rendered after the cacheable prefix.
- Rolling summary updated only at semantic seams, with explicit "preserve verbatim" rules and a word cap.
- Verbatim tail of three to six turns.
- Distilled tool results, with a retrieval path back to the full record.
- Retrieval over full history for identifier lookups and explicit references.
- Replay evals with planted facts and planted negations, run on every change to the trimming logic.
The parts that surprise people are 2 and 5. Extraction and distillation both look like extra work, and both remove the need for most of the trimming you were going to do anyway. Once facts live in state, the transcript is mostly context for tone, and you can cut it far more freely than you'd dare otherwise.
When not to trim at all
Trimming is not universally correct. Skip it when the conversation is short and bounded (under roughly ten turns), when the task is a single document transformation with no follow-up, or when the model is being used for a one-shot classification with a fixed prompt. In those cases, the machinery costs more in latency and complexity than it saves in tokens.
It's also the wrong tool when the real problem is that you're resending a huge static context — a long policy document, a schema dump, a codebase excerpt — on every turn. That's a caching and placement problem, not a trimming problem. Putting a 30,000-token document at the front of every request and then agonizing over 400 tokens of chat history is optimizing in the wrong place.
The question worth asking is not "how do I fit in the window." It's "which facts does this system actually need to be correct, and where do they live?" If the answer is "in the conversation," you have a design you can't test. If the answer is "in a state object I can inspect and assert on," trimming becomes an operational detail instead of a correctness risk.