How to Fix Multi-Agent LLM Latency: Text Handoffs Cost 2.5x

TL;DR Multi-agent LLM latency has a specific, measurable cause: pipelines that pass results between models as text are paying for a detour. An ICLR 2026 paper (Cache-to-Cache, or C2C) measured a 2.5x average latency penalty for text-based model-to-model communication versus letting models exchange KV-cache state directly, plus a 3.1-5.4% accuracy loss from the round-trip through natural language. You can’t deploy C2C against a closed API today, but the same paper is a precise measurement of a tax your own agent pipeline is already paying — and most of that tax is avoidable without touching a model’s internals.

A text handoff is what happens every time one LLM’s output becomes another LLM’s input in a multi-model system: the sender generates a token sequence, the receiver tokenizes it and re-encodes it from scratch. It’s the default in every popular agent framework, and it’s also the specific thing a new ICLR 2026 paper measured the cost of — precisely, in latency and in accuracy, not just in vibes.

What’s actually driving multi-agent LLM latency

Multi-LLM systems exist because one model rarely covers every specialization you need — a fast router, a careful reasoner, a code-specific model, a summarizer. The industry default for connecting them is text: model A produces an answer, model B reads that answer as its prompt. It’s simple, model-agnostic, and it’s also the paper’s stated motivation, because that simplicity has two real costs baked in.

First, generation is inherently sequential — the sending model can’t hand off its result until it has produced every token of it, one at a time, which is the latency floor for anything generated as prose rather than returned as a single structured object. Second, and more subtle, text is a lossy compression format for a model’s internal state. Whatever rich, high-dimensional representation the first model built up while solving its piece of the problem gets flattened into a sequence of words before the second model ever sees it — the second model has to reconstruct meaning from the words, not receive the representation directly.

Researchers at Tsinghua’s NICS lab asked the obvious follow-up question: if text is a bottleneck, what happens if two models skip it and talk cache-to-cache instead?

What is Cache-to-Cache (C2C) communication?

Cache-to-Cache (C2C) is a communication paradigm where one LLM’s KV-cache is projected and fused directly into a second LLM’s KV-cache by a small trained neural network, so the second model absorbs the first model’s internal state without either model generating or re-reading a single word of text.

The mechanism has two pieces. A projector network maps the source model’s cache representation into a form the target model’s layers can consume, and a learnable gate decides which of the target model’s layers actually benefit from that injected cache — not every layer needs the same amount of borrowed context. The paper’s own oracle experiments justified the approach before they built it: artificially enriching a model’s KV-cache with extra semantic content improved its response quality without growing the cache’s size, which told the authors that the cache itself, not just the tokens it was derived from, was a usable channel for meaning.

How much does a text handoff actually cost?

Benchmarked head-to-head against the same pair of models on the same tasks, C2C delivered an average 2.5x speedup in latency over text-based communication. That number is the direct payoff of skipping two sequential costs text pays and cache fusion doesn’t: token-by-token generation on the sending side, and a full forward pass to re-encode that generated text on the receiving side. Cache fusion replaces both with a single projection step against a representation the sending model already computed.

2.5x is an average across the paper’s benchmark suite, not a guarantee for your specific pipeline — a two-hop pipeline pays this tax once, a five-hop agent graph pays a version of it at every edge. That compounding is the part worth internalizing even if you never touch C2C’s code: the more hops your architecture has, the more this exact inefficiency multiplies.

Why does skipping text also raise accuracy?

The latency number gets the headline, but the paper’s accuracy result is the more interesting one for anyone building agent pipelines for correctness, not just speed. C2C reported 6.4-14.2% higher average accuracy than either model running alone, and — the comparison that matters here — 3.1-5.4% higher accuracy than the same two models communicating over text.

The mechanism is the same lossy-compression argument from the problem statement, just confirmed empirically: a model’s KV-cache carries “deep, specialized semantics” that a token sequence can’t fully capture. When the sending model writes out its answer in words, it’s summarizing its own internal state for a reader — and summaries drop information the original representation had. Cache fusion hands over the representation itself, so the receiving model works from strictly more signal than the text version of the same handoff gave it.

What can you do about this today?

You can’t drop C2C into a pipeline built on hosted APIs — it needs white-box access to both models’ KV-caches, and Anthropic, OpenAI, and Google don’t expose that over an API today. But the paper is a precise measurement of a cost your pipeline pays regardless, and most of the fix doesn’t require anyone’s cache at all:

1. Instrument every hop before you optimize any of them. Log tokens generated, wall-clock time, and dollar cost per model-to-model handoff. You cannot tell which hop is your 2.5x until you’ve measured each one separately.

2. Cut hops that exist only to re-explain, not to add expertise. If an orchestrator asks a worker for an answer, then asks a second model to “clean up” or “format” that answer in prose, you’ve paid for a full generation-and-re-encode round trip that added no new capability — merge it into the worker’s own output contract instead.

3. Replace prose handoffs with structured, schema-validated output. A worker returning {"result": ..., "confidence": ...} as tool-call output costs the receiving model a parse, not a re-derivation of meaning from natural language. This is the same principle behind writing agent tool instructions that shape output for the next consumer, applied to model-to-model handoffs instead of human-to-agent ones.

4. Reach for a second model only when its specialization earns the round trip. A hop to a different model is worth its latency when that model has real, narrow expertise the first one lacks — not as a default architecture pattern for every sub-task.

5. Use prompt caching to cut the cost of what you can’t eliminate, per Anthropic’s prompt caching guidance. It doesn’t remove a hop, but it removes the cost of re-processing a shared prefix across repeated calls to the same model — genuinely complementary to hop reduction, not a substitute for it.

6. Watch for cache-sharing APIs, don’t build around them yet. If a hosted provider ever exposes cache-level fusion between models, the calculus in this section changes. As of this ICLR 2026 paper, it hasn’t, so plan your architecture around what you can measure and change today.

Text handoff vs structured handoff vs cache-to-cache

Axis	Text handoff (prose)	Structured handoff (schema/tool output)	Cache-to-Cache (research)
What crosses the boundary	Generated natural-language tokens	A validated JSON/tool-call object	Projected, fused KV-cache state
Sequential generation cost	Full token-by-token generation	Shorter, schema-constrained generation	None — no intermediate text
Receiver’s decoding cost	Full re-encode of prose	Parse + light re-encode	Direct fusion into existing cache
Accuracy vs. text (paper’s numbers)	Baseline	Not benchmarked in this paper	+3.1-5.4%
Works with closed model APIs today	Yes	Yes	No — needs white-box cache access
What it takes to adopt	Nothing — it’s the default	A shared schema both sides honor	A trained projector per model pair

Read this table as a migration path, not a ranking: almost every pipeline can move from column one to column two this week. Column three is where the field is heading, not where production agent systems can go yet.

What breaks if you ignore the handoff tax?

Nothing breaks loudly, which is exactly the danger. A pipeline with unnecessary text hops still works — it’s just slower and more expensive than it needs to be, and that cost hides in aggregate latency and API bills rather than in an error you’d notice and fix. Left alone, it compounds as you add more specialized models: each new hop you bolt on for “one more capability” adds its own generate-then-re-encode tax, and a five-model pipeline can end up paying that tax more times than the actual reasoning it does would justify.

The paper’s 2.5x number is also a ceiling on how much headroom exists in the communication layer specifically — it says nothing about hops that are actually necessary because a second model’s expertise is genuinely required. Chasing the number by merging every hop indiscriminately trades latency for capability you were paying for on purpose. The fix is measuring which hops are real work and which are re-explaining, not eliminating hops as a blanket rule.

Common mistakes when building multi-agent pipelines

Adding a model instead of a rule. A round trip to a second model is a bigger latency and cost commitment than a conditional in the orchestrator. Reach for it only when the specialization is real.

Letting “clean up the output” become its own hop. Formatting, summarizing, or restating another model’s answer in prose is the purest form of the text tax this research is about — it adds a full generate-and-re-encode cycle for zero new information.

Benchmarking a pipeline change on cost per call instead of cost per hop. A cheaper model at the same number of hops still pays the same multiplicative tax; the hop count is usually the bigger lever than the per-call price.

Assuming prompt caching solves this. It cuts the cost of re-processing a repeated prefix on one model, not the cost of one model’s output becoming another model’s input. They’re complementary, not the same fix.

FAQ

What is Cache-to-Cache (C2C) communication between LLMs? C2C is a research paradigm from an ICLR 2026 paper where two LLMs exchange information by projecting and fusing their KV-caches directly, instead of one model generating text that the other re-reads. A small neural network and a learnable gate decide which of the target model’s layers should receive the source model’s cache.

How much latency does text-based multi-agent communication actually cost? The C2C paper measured an average 2.5x latency speedup for cache-based communication over text-based communication between the same pair of models, on the same tasks. That gap comes from token-by-token generation on the sending side plus a full re-encoding pass on the receiving side.

Can I use Cache-to-Cache in production today? Not as a drop-in library. It needs a projector network trained for a specific pair of models with white-box access to both models’ KV-caches, which rules out closed APIs like hosted Claude, GPT, or Gemini. The code is public for open-weight model pairs, but it’s a research direction to watch, not infrastructure to deploy this quarter.

Does reducing text handoffs also improve accuracy, not just speed? Yes. C2C reported 3.1-5.4% higher accuracy than text-based communication between the same two models, and 6.4-14.2% higher than either model running alone, because the receiving model gets the sender’s richer internal representation instead of a lossy text summary of it.

What can I do right now to cut multi-agent latency without waiting for cache-sharing? Count your hops and cut the ones that only re-explain a result in prose, replace free-text handoffs with structured, schema-validated output, and reach for a second specialized model only when its narrow expertise earns a full round trip. None of that needs a projector network.

Does this apply if I’m using closed-source model APIs like GPT or Claude? Cache-to-Cache itself does not, since it needs raw KV-cache access neither vendor exposes. The underlying lesson still does: every hop where one closed model’s output becomes another’s input pays full generation-and-re-encoding cost, so the structured-output and hop-count fixes apply regardless of which models you’re calling.

Sources

• Cache-to-Cache: Direct Semantic Communication Between Large Language Models — Tianyu Fu, Zihan Min, Hanling Zhang, Jichao Yan, Guohao Dai, Wanli Ouyang, Yu Wang. ICLR 2026. The primary source for the 2.5x latency speedup, the 3.1-5.4% and 6.4-14.2% accuracy figures, and the projector-plus-gating architecture.
• thu-nics/C2C on GitHub — the paper’s public code, for open-weight model pairs.
• Anthropic — Prompt caching — the production-available mechanism for cutting repeated-prefix cost within a single model, referenced in the “what to do today” section.

The takeaway

The paper’s 2.5x number is a research result you can’t deploy against a hosted API today, but it’s also the most precise measurement anyone has published of a cost every multi-agent pipeline already carries. You don’t need a trained projector network to act on it — you need to count your hops, cut the ones that only re-explain, and replace the rest with structured output instead of prose. Do that first; watch cache-sharing APIs second.

If you’re already deep in agent-pipeline cost work, cutting agent tool call cost covers the same instinct applied inside a single agent’s tool loop rather than between two models, and running agent tool calls in parallel is the complementary fix for pipelines where the hops are necessary but currently sequential. For the mechanics of what a KV-cache actually holds and costs, see the VRAM math behind a model’s KV-cache, and for the token-budget side of the same problem inside one long-running agent, see what survives agent context compaction.

Explore more: LLM Engineering