Agent Harness Design: Why an ARC-AGI-3 Score Tripled

A benchmark score moved from 13.3% to 38.3% without the model changing at all. Same weights, same reasoning budget, no task-specific tools. The only variable was agent harness design — the code around the model that decides what it sees on each turn.

That is the whole story of OpenAI’s ARC-AGI-3 harness post, published in late July 2026, and it is the most useful benchmark controversy of the year — not because of who is ahead, but because it puts a number on something most teams building agents have never measured: how much performance their own harness is throwing away.

If you run a multi-turn agent in production, there is a good chance you are leaving a comparable multiple on the table right now, for exactly the two reasons OpenAI names.

TL;DR

Agent harness design is the code between your model and its environment — formatting, history, reasoning persistence, context eviction. It is the part you write, and it caps what the model can do.
OpenAI reports GPT-5.6 Sol going from 13.3% → 38.3% on the ARC-AGI-3 public set with no model change, by enabling two settings: retained reasoning and compaction.
Retained reasoning stops the agent re-deriving its own conclusions every turn. Compaction replaces blind truncation with summarization. Together OpenAI also reports ~6× fewer output tokens.
ARC Prize is not wrong to refuse this. A leaderboard has to hold the harness constant or it stops comparing models. Both parties are measuring real, different things.
The takeaway is not the leaderboard. It’s that you should diff turn N against turn N+1 in your own agent today and find out what you’re deleting.

What is agent harness design?

Agent harness design is the practice of engineering the code that mediates every exchange between a model and its environment: it decides how observations are rendered into text, what survives from previous turns, whether intermediate reasoning is carried forward, and what gets evicted when the context window fills. The weights are frozen and vendor-supplied. The harness is yours.

Most teams don’t think of it as a design surface. They think of it as glue — a while loop, a list of messages, an append. That framing is how a system quietly loses two-thirds of its capability without a single failing test.

ARC-AGI-3 makes the cost unusually visible because it is an interactive benchmark. ARC Prize describes it as challenging agents “to explore novel environments, acquire goals on the fly, build adaptable world models, and learn continuously.” There are no instructions. The agent takes an action, observes a grid, forms a theory, tests it, and revises. Every one of those steps depends on the previous ones. A harness that forgets is not slightly worse at this task — it is categorically unable to do it.

The numbers

Configuration	ARC-AGI-3 public set	Source
GPT-5.5, official harness	0.4%	ARC Prize leaderboard
GPT-5.6 Sol, official harness	7.8%	ARC Prize leaderboard
Claude Opus 5, official harness	30.2%	ARC Prize leaderboard
GPT-5.6 Sol, OpenAI’s baseline run	13.3%	OpenAI
GPT-5.6 Sol, retained reasoning + compaction	38.3%	OpenAI
Human testers (average)	~48%	ARC Prize

Two things in that table deserve more attention than the headline.

First, the honest gap: OpenAI’s own reproduction of the official harness scored 13.3%, not the 7.8% on ARC Prize’s board. That is a 5.5-point discrepancy in a baseline that both sides believe is the same configuration, and it is unexplained. The tripling claim is measured against OpenAI’s own baseline, not the public one — a detail that gets lost in every summary of this story.

Second, 38.3% is self-reported, on the public set, and has not been independently replicated. The Decoder’s write-up is right to frame it as a claim rather than a result. Treat it as a strong signal about harness sensitivity, not as a settled ranking.

The signal survives both caveats. Whatever the exact baseline, a harness change moved the number by more than the gap between two model generations.

Retained reasoning: your agent restarts its own thinking every turn

Here is the failure mode in one sentence. In a standard multi-turn loop, the model thinks, emits an action, and the thinking is thrown away. The next turn begins with the visible transcript — the moves and their outcomes — but not the reasoning that produced them.

So the agent sees that it moved left and the grid changed, but not why it believed moving left would test the hypothesis it was testing. It re-derives the theory from the evidence, every single turn, and there is no guarantee it derives the same one twice. An exploratory task becomes a sequence of independent guesses by an amnesiac that keeps finding its own notes.

Retained reasoning fixes this by persisting the model’s reasoning items across turns rather than discarding them at the action boundary. On OpenAI’s Responses API, that means chaining turns with previous_response_id instead of hand-assembling a message array — the reasoning is preserved server-side and carried forward.

This is also where the ~6× output-token reduction comes from, and it is worth understanding why a more stateful setup is cheaper. Re-deriving a conclusion costs reasoning tokens every time. Carrying it forward costs them once. The efficiency gain isn’t a separate optimization; it’s the same fix seen from the billing side.

The generalizable lesson has nothing to do with OpenAI’s API: if your agent’s turn N+1 payload does not contain turn N’s reasoning, you are paying for that thinking repeatedly and getting a different answer each time. Most hand-rolled loops built on chat-completions-style APIs strip reasoning by default, because the API shape encourages you to rebuild the message list yourself.

Compaction: truncation is a silent lobotomy

The second setting addresses what happens when context runs out.

The default behavior in most harnesses is rolling truncation: drop the oldest messages until the payload fits. It is one line of code and it is the worst possible eviction policy for an exploratory agent, because in exploration the oldest observations are the most load-bearing ones. Turn 3 is where you learned the rule. Turn 90 is where you’re applying it. Truncation deletes turn 3.

Compaction replaces the oldest turns with a condensed representation of what they established, rather than deleting them. OpenAI’s implementation triggers around a 175,000-token threshold. The conclusions survive; the transcript doesn’t.

The tradeoff is real and worth naming: compaction is lossy in a way you cannot inspect. A truncated context is at least legible — you know exactly what is gone. A compacted one has been summarized by a process whose omissions are invisible until the agent acts on something it no longer knows. You are trading a known, catastrophic loss for an unknown, smaller one. That is usually the right trade, and it is still a trade.

If you are building a compaction layer yourself, the useful discipline is to summarize conclusions and constraints explicitly rather than asking for a generic summary. “Rules established so far” and “hypotheses ruled out” are worth ten times their token cost. A prose recap of what happened is worth almost nothing.

ARC Prize is also right

It would be easy to read this as OpenAI exposing a flawed benchmark. It isn’t, and the counter-position is straightforward.

ARC Prize runs a standardized harness — no provider-specific settings — because a leaderboard’s entire job is comparison, and comparison requires holding everything except the variable constant. The moment each vendor ships its own harness, the ranking measures vendor engineering effort as much as model capability, and “Model A beats Model B” stops carrying information.

So both claims are true at once:

ARC Prize measures models. Under identical conditions, Claude Opus 5 scored 30.2% and GPT-5.6 Sol scored 7.8%. That comparison is valid and it is the number a leaderboard should publish.
OpenAI measures a product. Nobody ships the standardized harness. Users touch ChatGPT and Codex, which run the retained-reasoning-and-compaction configuration. That number is also meaningful, and it isn’t a leaderboard number.

The mistake is demanding that one number do both jobs. This is the same reason evals you run on your own harness beat public benchmarks for deciding what to ship: a leaderboard answers “which model,” and you are almost never asking that question. You are asking “does my system get better,” and only your harness can answer it. It’s also why model-release benchmark tables should be read as a floor on capability, not a prediction of what you’ll observe.

What to actually do about it

Four things, in order of how much they will pay you back.

Diff your turns. Log the full input payload for turn N and turn N+1 of a real multi-turn run and compare them. If turn N’s reasoning or tool-call rationale is missing from N+1, that’s the finding. This takes twenty minutes and is the only step that is genuinely non-optional.
Find your eviction policy. Read the code — or the framework’s docs — for what happens at the context limit. If the answer is “drops the oldest messages,” you have the truncation problem. Replace it with summarization that preserves established facts and ruled-out branches.
Use the stateful path your provider offers. If your API has a response-chaining primitive that preserves reasoning, use it instead of rebuilding the message array. Hand-rolled history management is how reasoning gets silently dropped.
Put the harness under test. Version it, and re-run your eval suite when it changes. A harness change is a deploy with the same blast radius as a prompt change — and considerably more than a model upgrade, on this evidence.

Where teams get this wrong

The most common error is treating harness work as plumbing to be finished rather than a component to be owned. Symptoms: no version number, no tests, no owner, and a while loop that has not been read since the prototype.

The second is reaching for a bigger model when the agent behaves inconsistently across turns. Inconsistency across turns is the signature of state loss, not weak reasoning — and upgrading the model makes the discarded reasoning more expensive, not less. This is the same trap as shipping a demo and calling it production: the prototype worked because the runs were short enough that nothing had been evicted yet.

The third is over-correcting into stuffing everything into context forever. Compaction exists because unbounded context is neither affordable nor effective. The goal is to keep the conclusions, not the log.

The take

The interesting claim in this story is not 38.3%. It’s that the gap between a naive harness and a considered one was larger than the gap between two model generations of the same family — and that nobody found it until a vendor went looking for a leaderboard argument.

Your agent’s harness is not glue code. It is the component that decides what your model knows, and on current evidence it may be the highest-leverage code in your stack. Most teams have never profiled it. Go diff two consecutive turns and find out what you’ve been deleting.

If you’re building the surrounding infrastructure next, designing agent-to-human handoffs and standing up an MCP server are the two places this same “what does the model actually see” question shows up again.

FAQ

What is an agent harness? An agent harness is the code around the model that decides what the model sees on each turn: how observations are formatted, which parts of the history survive, what happens to intermediate reasoning, and when older context is dropped or summarized. The model weights are fixed; the harness is the part you actually write. Because it controls the input, it sets a ceiling on performance that no amount of model capability can lift.

What is retained reasoning and why does it matter? Retained reasoning means carrying the model’s internal reasoning items forward into the next turn instead of discarding them once an action is emitted. Without it, a multi-turn agent starts every turn from the visible transcript alone, re-deriving hypotheses it already formed and paying for that thinking again. On OpenAI’s Responses API it is enabled by chaining turns through previous_response_id rather than rebuilding a message array by hand.

What is context compaction? Compaction places a condensed summary of older turns into the context window when it fills, instead of hard-truncating the oldest messages. Truncation is lossy in the worst possible way — it silently deletes the earliest observations, which in an exploratory task are usually the most load-bearing ones. Compaction keeps the conclusions and drops the transcript.

Did OpenAI’s model get better at ARC-AGI-3? No. OpenAI reported the same model, no additional training, no larger reasoning budget, and no task-specific tooling. The only thing that changed was the harness that mediated between the model and the benchmark environment, which is precisely why the result is interesting to engineers rather than to model researchers.

Why does ARC Prize use a standardized harness instead of each vendor’s best setup? Because a leaderboard is a comparison, and a comparison requires holding everything except the model constant. If every vendor supplies its own harness, the number measures the vendor’s engineering effort as much as the model, and cross-model rankings stop meaning anything. Both positions are defensible: ARC Prize is measuring models, OpenAI is measuring products.

How do I know if my own agent has this problem? Log the exact input payload sent on turn N and turn N+1 of a real multi-turn run and diff them. If turn N’s reasoning or tool-call rationale is absent from turn N+1, you are discarding it. Then check what your framework does at the context limit — if the answer is “drops the oldest messages,” you are truncating, not compacting.

Sources

OpenAI, “How enabling two settings tripled our scores on the ARC-AGI-3 benchmark” — the 13.3% → 38.3% figures, the two settings, and the ~6× output-token reduction.
ARC Prize, ARC-AGI-3 — the benchmark’s design goals and the standardized-harness position.
The Decoder, “OpenAI claims GPT-5.6 Sol beats Opus 5 on ARC-AGI-3 with its latest API and two additional settings” — the official-harness comparison numbers and replication status.

Written for umesh-malik.com — no-fluff technical writing on AI, Web Dev, and Engineering.