Fix Agent Prompt Cache Misses: A 14% First-Request Hit Rate

TL;DR — Agent prompt cache misses usually start on the very first request, where one system-prompt block mixing shared and per-job text makes every session hash differently. Cloudflare found its reviewer’s first request hit the cache only 14% to 26% of the time, split the prompt at a marker so the shared 11k-21k tokens got their own breakpoint, and cut the average review from $2.43 to $0.77 alongside other changes. Log your own first-request hit rate before you touch anything else.

A prompt cache miss is a request whose input prefix was not found in the provider’s cache, so the model re-reads and re-bills tokens it has already seen at the write price instead of the read price.

What Is a Prompt Cache Miss on the First Request?

Prompt caching stores the already-computed prefix of a prompt and reuses it when a later request starts with exactly the same bytes. Cloudflare’s write-up on cutting its AI reviewer’s inference bill by 68% gives the pricing that matters: a cache write costs 1.25 times the normal input price, a cache read costs 0.1 times, and an entry expires after five minutes unless something reads it.

Two facts follow, and both are easy to miss:

• Only a prefix is cached. Anything that changes in the middle or at the end of the conversation cannot be reused. The system prompt and tool definitions are the ideal candidate because they rarely change.
• A breakpoint decides what gets stored. The prefix is saved where a cache breakpoint ends it. Put the only breakpoint after a block that contains per-job text and the shared part is never stored on its own.

That second one is the trap. Inside one session the cache works fine, because call two reuses what call one wrote. Across sessions it silently fails, and your bill grows without any single request looking wrong. I covered the in-session side of the same cost curve in why a better tool set raised agent cost until the prompt was rewritten.

How Did Cloudflare Find a 14% First-Request Hit Rate?

Cloudflare’s average review had crept from $1.19 in April to about $2.46 by the end of August with no architecture change. Token spend broke down as output 23%, cache writes 43% and cache reads 34%, so most of the money went on context the model had already seen or was about to see again.

Starting September 14, the reviewer logged two things for the first LLM call of every session: its token usage, and a hash of the first system block exactly as sent to the provider. The result was blunt. The first request found a cached prefix only 14% to 26% of the time, even though every session opened with the same tool definitions and agent prompt, between 11k and 21k tokens identical in every repository.

The cause was how the harness built the system prompt. It emitted one block holding the agent prompt followed by an environment section with the working directory and the date, and placed the only breakpoint at the end. Every repository and every day produced a different block, so one review could never read what another had written.

How Do You Audit Your Own First-Request Hit Rate?

You need only the usage object the API already returns. On Anthropic’s API each response reports cache_creation_input_tokens and cache_read_input_tokens; log them with the session id, the call index and a hash of the first system block. Then count how often call 1 of a session read more than it wrote.

I wrote a 30-line Node script for that and ran it on a synthetic 200-session log, where it reported a 20.5% first-request hit rate and 41 distinct first-block hashes, which is the signature of a prefix that varies per job.

import { createReadStream } from "node:fs";
import { createInterface } from "node:readline";

// One JSON object per line: { session, call, prefixHash, usage }
const firstCalls = new Map();
for await (const line of createInterface({ input: createReadStream(process.argv[2]) })) {
  if (!line.trim()) continue;
  const row = JSON.parse(line);
  if (row.call === 1) firstCalls.set(row.session, row);
}

Then score each session’s first call and group by prefix hash:

let hits = 0;
const byHash = new Map();
for (const { prefixHash, usage } of firstCalls.values()) {
  const read = usage.cache_read_input_tokens ?? 0;
  const write = usage.cache_creation_input_tokens ?? 0;
  const hit = read > 0 && read >= write;
  if (hit) hits++;
  const h = byHash.get(prefixHash) ?? { sessions: 0, hits: 0 };
  h.sessions++;
  if (hit) h.hits++;
  byHash.set(prefixHash, h);
}

console.log(`first-request hit rate: ${((100 * hits) / firstCalls.size).toFixed(1)}% of ${firstCalls.size} sessions`);
console.log(`distinct first-block hashes: ${byHash.size} (should be 1 per prompt version)`);

Read the two lines together. A low hit rate with many hashes means the prefix is varying. A low hit rate with one hash means something else: traffic too sparse for the five-minute window, or parallel starts racing each other.

How to Fix Prompt Cache Misses in 5 Steps

1. Log the first request. Record usage and the hash of the first system block for call 1 of every session, as above. Without this you are tuning blind.

2. Put a marker at the end of the shared text. Cloudflare appended a marker to every agent prompt and used a plugin hook to split the block in two at that marker. The first half ends on bytes every job shares.

3. Give the shared half its own breakpoint. The environment section, working directory and date go in the second block, after it.

4. Remove anything job-specific from in front of the marker. Cloudflare disabled the list of available skills and froze tool descriptions between jobs, because either one changes the bytes in front of the breakpoint.

5. Stagger parallel starts. A cache entry cannot be read until the response that wrote it has started streaming. For three reviewers sharing a prefix, Cloudflare has the second and third wait up to 20 seconds so one writes and two read, which costs a little latency and saves about nine cents per reviewer.

Then re-run the audit. The target is a hit rate near 100% on call 1 and one hash per prompt version.

If your agent reads reference files in its first turns, fold them into the shared prefix too. Cloudflare moved its standards reviewer’s reference index into the prompt; LLM calls per run dropped from 15.9 to 9.6 and cost per run from $1.12 to $0.72. Context you re-fetch every session belongs in the cached prefix, not behind a tool call. The same logic applies to your CLAUDE.md: keep it stable and short, because it is part of the prefix.

How Much Does a First-Request Miss Cost?

I modelled the first request only, in input-token equivalents (one uncached input token = 1), using the 1.25 and 0.1 multipliers and a 16k-token shared prefix, the midpoint of Cloudflare’s 11k-21k range. The numbers are arithmetic on those assumptions, not a measurement of anyone’s production traffic.

A miss is 12.5 times as expensive as a hit on the same tokens (20,000 versus 1,600 for 16k), so the gap between a 14% and a 97% hit rate is about 8x on the prefix alone. At an assumed $3 per million input tokens, 1,000 sessions a day at 20% hits costs about $49 for first-request prefixes against $6.46 at 97%.

The split also pays for the part that still varies. With a 300-token environment block, a single block costs 20,375 per first request; a split block costs 1,600 for the cached read plus 375 for the new write, 1,975 in total.

What Else Cut Cloudflare’s Bill From $2.43 to $0.77?

Prefix sharing was the biggest single saving, but not the only one. Cloudflare attributes $0.65 of the $1.67 saving to code changes and $1.06 to model and effort changes, measured as weekday hours before and after each change, which it notes is not a controlled experiment.

Fewer calls compound the cache win, because each call resends the whole conversation. If you are also trimming context mid-session, what survives agent context compaction shows what you can lose without breaking the run.

What Breaks If You Cut Effort to Save Money?

Cheaper is not automatically fine. Cloudflare checked quality two ways: whether authors changed the flagged lines, and a blinded judging pass over 657 findings. For the code quality reviewer, authors acted on 34% of serious findings before and 59% after, and important findings per review went from 0.28 to 0.30. What vanished was mostly minor findings and inflated severity.

Medium effort was not free. Code quality at medium led to roughly a quarter as many acted-on findings as at high, so Cloudflare moved it back to high, which cost about 61% more per run. The performance reviewer lost about 64% of important findings, around one fewer per 24 full-tier reviews. The windows are short and samples are 35 to 66 findings per group, so treat these as directional.

Fix the cache first because it costs you no quality at all. Touch model and effort second, with an outcome metric attached. For reviewers specifically, routing changes by risk tier tells you which PRs deserve high effort, and Anthropic’s code review guide covers the reviewer setup itself. Latency tells the same story: passing text instead of full context between agents cuts tokens at a different layer. Related topic hubs: Claude Code and LLM engineering.

FAQ

Why does my agent miss the prompt cache when the system prompt looks identical?

A cache entry only matches on an exact byte-for-byte prefix. If anything that varies, such as a working directory, a date, a skill list or a tool description, sits in front of the cache breakpoint, every request hashes differently and nothing is reused. Move the varying text after the breakpoint.

How much more does a cache write cost than a cache read?

On Anthropic models a cache write costs 1.25 times the normal input price and a cache read costs 0.1 times, per Cloudflare’s write-up. A miss on the first request is therefore 12.5 times as expensive as a hit on the same tokens.

How long does a cached prefix last?

An entry expires after five minutes unless something reads it, which refreshes it. A low-traffic agent that runs once every ten minutes will miss every time, no matter how clean its prompt is.

Can parallel agents share one cache entry?

Yes, but not if they all start at the same instant. An entry cannot be read until the response that wrote it has started streaming, so Cloudflare lets one reviewer write and makes the other two wait up to 20 seconds before sending the same first request.

Does lowering reasoning effort to save money hurt quality?

Sometimes. Cloudflare’s code quality reviewer kept its findings at high effort, but medium effort led to roughly a quarter as many acted-on findings, and its performance reviewer lost real findings at medium. Measure outcomes per reviewer before you cut effort.

Sources

• Cutting our inference bill by 68% by being obsessive about optimization, Cloudflare Blog. Source of every Cloudflare figure above.
• Prompt caching, Anthropic documentation. Cache breakpoints and the usage fields.
• Cost table and audit script: my own arithmetic and code, using the multipliers Cloudflare quotes; the script was run on a synthetic log, not production data.