How to Review AI-Written Code You Don't Understand

TL;DR — How to review AI-written code you don’t fully understand: stop asking “did a human read it?” and ask what could go wrong, and which check catches it. Sort every change into three risk tiers by the paths it touches: low merges on green automated checks, medium gets one person, high always gets a person plus a rollback plan. I ran that rule over this repo’s 476 commits and 54% landed in the low tier, which is the share of review time you can safely hand to machines.

Risk-tiered review is a policy that routes each change to the cheapest check that can actually catch its failure mode, instead of sending every change to a human by default.

What Is Risk-Tiered Code Review?

A Hacker News thread titled Ask HN: How to deal with shipping AI-written code you don’t understand? hit 239 points and 118 comments, and the answers split into two camps. One says stop shipping what you can’t explain. The other says you never read your colleagues’ code line by line either, so tests and trust have to carry it.

Both are half right, and the Claude Code team gave the cleanest resolution I have seen. In How the Claude Code team does code review, Addy Osmani, Morgan Westlee Lunt and Qing Wang write that “did a human read the code?” is an imprecise proxy for “what could go wrong with this change, and which check, automated or human, is the right one to catch it?” Simon Willison pulled out that exact quote.

That reframing is the whole idea. You stop asking whether a person looked and start asking what a person would catch that a test would not. The answer is rarely “nothing” and rarely “everything”.

Question	Old habit	Tiered approach
Who reviews?	A teammate, always	Depends on what the change touches
What is the gate?	A human approval	The check that fits the failure mode
What does the author prove?	“It works on my machine”	Output from tests, build and lint, shown
Who owns the outcome?	The approver	The person directing the agent

How Do You Sort a Change Into a Risk Tier?

The rule that matters most in Anthropic’s write-up is this one: the change sets the tier, not the author. An agent’s approval never counts as a person’s, and the person directing an agent counts as a required reviewer. Their published tiers look like this:

Tier	Examples	Human approvals
Low	Docs, copy, internal-only features, additive changes behind a flag	None, once automated review and security checks are green
Medium	Ordinary product changes, plus sensitive areas such as caching	One person
High	Billing, the auto-updater, CI workflow files	Two people

I wanted to know what this looks like on a real, small codebase, so I wrote a 20-line script that classifies every non-merge commit in this repo by the paths it touched. A commit takes the highest tier of any file in it. The rules were deliberately blunt:

• High: .github/workflows/, the Cloudflare Worker, wrangler config, _headers and _redirects, the lockfile.
• Low: Markdown, blog covers and diagrams, docs, the content log.
• Medium: everything else, meaning application source.

Across 476 commits from 2026-02-08 to 2026-10-19, the split was 255 low (53.6%), 146 medium (30.7%) and 75 high (15.8%).

Two caveats keep this honest. It counts commits, not lines, and a path rule is a crude stand-in for a real classifier. This repo is also content-heavy, since an unattended job publishes posts, so your low share will be smaller. The point is the shape: more than half of changes carry almost no blast radius, and 1 in 6 touches something that can take the site down.

How to Review AI-Written Code in 6 Steps

This is the procedure I now apply to any change an agent wrote, whether or not I understand every line.

1. Write the failure sentence. One line: “If this is wrong, then ___.” If you cannot finish it, you do not yet know the tier, so treat it as medium or higher.

2. Classify by path. Use a table like the one above, stored in the repo. Anything touching CI, deploy config, auth, payments, secrets or data migrations is high, automatically.

3. Make the agent show its evidence. Require pasted output from the test run, the build and the linter. “It should work” is not evidence. Boris Cherny’s estimate in the Anthropic piece is that giving Claude a way to verify its own work makes the result two to three times better.

4. Get an independent check. One commenter on the HN thread, edwin0, named the real trap: when the AI writes both code and tests, they can agree with each other and still miss the same thing. Break the symmetry with a fresh-context reviewer, mutation testing, or a runtime assertion you wrote.

5. Pin approval to one commit. An approval covers the commit that was reviewed. New commits the system cannot vouch for need re-approval, otherwise a green check on commit one blesses whatever lands on commit nine.

6. Ship the risky ones reversibly. Put new behavior behind a flag, release to a small group first, and know the rollback command before you merge. Observability does the work reading cannot. As billylo put it in the thread, add extra runtime cross-checking and alerting to offset the risk.

None of this requires understanding the diff line by line. It requires understanding the blast radius, which is a much smaller thing to hold in your head.

What Breaks If You Skip the Tiers?

The failure is not the one people fear. It is not that an AI writes a subtle bug. It is that you spend scarce human attention in the wrong place, in both directions.

On the over-review side, Anthropic reports that about 59% of what their reviewer kept flagging was code that had been there since its first pass, and one CLI pull request went through more than fifty review rounds overnight before its author stopped it. The fix was a three-round cap, diff-only re-review after the first few passes, and flagging the bug class instead of a single line. With those, the share passing in one round rose from 65% to 78%, and the share needing five or more rounds fell from 8.9% to 1.5%.

On the under-review side, you merge a CI workflow edit on a green check. That is the file that decides what runs with your secrets, and it is exactly the kind of change a test suite cannot judge. Anthropic’s rule is that workflow files and owner-listed sensitive paths always need a person.

The other failure is quieter: nobody owns the outcome. Their line is “you land it, you own it”, and they add that it is not about blame. If you direct an agent, you are the reviewer of record, even when you skimmed the diff, which is why the evidence in step 3 matters.

When Is Skipping Human Review Actually Fine?

Skipping a human read is fine when three things hold at once: the change is low tier, the automated checks are green and independent of the author, and the failure would be visible and cheap to reverse. Docs, copy and additive code behind a flag fit. For a compiler author on that HN thread, a gate that takes hours across five architectures plus a large test corpus is a legitimate stand-in for reading every line.

It is not fine when the check shares the author’s blind spots, or when the change can fail silently. A wrong paragraph is noticed by a reader. A wrong permission in a workflow file is noticed by an attacker.

If you are building the policy for a team, start small. Put the path table in the repo, require the six steps on high-tier paths only, and measure human approvals per pull request and the revert rate. Anthropic uses revert and incident rates as the guardrail, and notes they have no circuit breaker yet that tightens approval automatically when those numbers rise. Build that one early.

For deeper dives on the pieces around this, see AI-generated code guardrails, Anthropic’s code review in Claude Code, grounding agent findings in evidence, approval fatigue in agent permissions, when to give AI agents autonomy and spec-driven development with agents. The wider cluster lives on the AI coding agents hub.

FAQ

Is it ever OK to ship AI-written code you don’t fully understand?

Yes, when the change is low risk and a check you trust can fail loudly if it is wrong. Docs, copy and additive changes behind a flag qualify. Billing, auth, deploy pipelines and anything that touches secrets do not, and a person who understands them has to sign off.

What is the biggest risk when an AI writes both the code and the tests?

The two can agree with each other and still miss the same thing, because they share one set of assumptions. One Hacker News commenter put it as ‘how do you check without doing the whole job twice’. The practical answer is an independent check: a fresh-context reviewer, mutation testing, or a runtime assertion the author did not write.

Does ‘a human read the code’ still matter?

It matters for the right changes. The Claude Code team’s position is that it is an imprecise proxy for a better question: what could go wrong with this change, and which check is the right one to catch it. For some changes that check is a person, for others it is a test suite.

How do I decide the risk tier of a pull request?

Decide it from what the change touches, not from who or what wrote it. Paths such as CI workflows, deploy config, auth and payments are high. Content and docs are low. Everything else is medium and gets one reviewer.

How many review rounds should an AI reviewer get?

Cap them. Anthropic’s team caps at three rounds, then the author reconsiders the approach, because one CLI pull request went through more than fifty overnight rounds. After a few passes, re-review only the diff and flag the bug class rather than one line.

Sources

• Addy Osmani, Morgan Westlee Lunt and Qing Wang, How the Claude Code team does code review: tiers, convergence numbers, quotes.
• Simon Willison, Quote from the Claude Code review post, 16 October 2026.
• Hacker News, Ask HN: How to deal with shipping AI-written code you don’t understand?, 239 points, 118 comments.
• First-party measurement: path-based classification of 476 non-merge commits in this repository, 2026-02-08 to 2026-10-19.