How to Scope an AI Agent for Reverse Engineering: 4 Weeks, Not Years

TL;DR A four-week Linux GPU driver for Apple’s M4 silicon shows how to scope an AI agent for reverse engineering: hand it one failing test and a narrow reference trace, not a full specification, and only widen the target once that one case passes. That single ordering change — concrete test first, exhaustive spec never — is why work that normally takes engineers years shipped in weeks. The same pattern generalizes to any hard technical task you hand an LLM coding agent, from firmware to proprietary protocols.

Test-first scoping is the practice of giving an LLM coding agent one concrete, checkable artifact to satisfy — a single failing test, a single byte-accurate reference — instead of a complete specification, and only broadening the target once that first case passes. It sounds like a small ordering trick. In one recent project it was the difference between a graphics driver that normally takes years and one that shipped, fully conformant, inside a single month.

What is test-first scoping for an LLM agent?

Most people scope an agent’s hard task the way they’d scope a human’s: hand over the documentation, the spec, the reference manual, and ask it to implement the thing. That feels thorough. It is also exactly backwards for reverse engineering, where the “documentation” is usually incomplete, wrong in places, or doesn’t exist at all — Apple never published a spec for the M4’s GPU firmware.

Test-first scoping flips the order. Instead of “read everything, then build,” the loop is “pick the smallest slice that can pass or fail, get it passing, then pick the next slice.” The agent never holds more of the problem in its head than the one thing it’s currently being judged against, and every iteration produces a real signal instead of a guess.

Why reverse-engineering tasks break generic agent workflows

Generic coding-agent guidance assumes a spec exists somewhere the agent can read: a language reference, an API doc, a working example to imitate. Reverse engineering removes that assumption entirely. There is no ground truth to read — only a black box you can observe and a reference implementation you can compare against, if you’re lucky enough to have one.

Handed a task like that with spec-first instructions, an agent does what spec-first instructions always produce: it explores broadly, infers structure from incomplete signals, and implements a large surface area before anything gets checked. Every one of those inferences compounds. By the time the agent finally runs its code against real hardware, it can be wrong in a dozen places at once, and there’s no way to tell which guess broke which behavior.

That’s the failure mode a real project ran into building a GPU driver for the M4 Mac Mini: the A18 Pro’s firmware ABI carries roughly 1.5x more structs and 2x more pointers than the earlier M1/M2 generation the team had prior experience with, and a captured compute workload came in at 336 MB — large enough that replaying it directly for feedback was, in the team’s own words, initially impossible. A spec-first attempt at that surface has nowhere to get its first real signal.

How a four-week GPU driver rebuilt the case for test-first scoping

The project — a fully OpenGL ES 3.0-compliant driver for the M4 Mac Mini and MacBook Neo, built atop Mesa’s Gallium abstraction layer — didn’t start by trying to document Apple’s AGX GPU instruction set exhaustively. It started with hardware tracing through a hypervisor, capturing what the firmware actually did without ever opening an Apple binary, and then handed the coding agent (Codex) one narrow translation task at a time: take this piece of Mesa’s NIR intermediate representation, produce the AGX instructions that make this one captured trace replay correctly.

That one-piece-at-a-time approach is what the team called “Mesa-first”: build only the features Mesa’s own test suite demanded next, instead of exhaustively documenting the hardware before writing a line of driver code. It outperformed the hardware-first alternative because a concrete, narrow objective focuses an agent’s effort; a theoretical completeness goal gives it nowhere to stop guessing. The Khronos OpenGL ES conformance test suite (the CTS) supplied the widening surface once Mesa-first got individual pieces working — a standing, ever-growing set of pass/fail checks the driver had to keep satisfying as scope expanded.

The result: a driver capable of running Minecraft at 212 fps on the M4 Mac Mini, built in about four weeks — not the days the team initially hoped for, but nowhere near the years this class of reverse-engineering project usually takes when it’s spec-first and unassisted.

How to scope an AI agent for reverse engineering in 6 steps

The GPU driver project and an earlier one on this site — reverse-engineering a proprietary printer language to fix a macOS driver HP never shipped — used the identical shape, just at very different scales. Both reduce to the same six steps:

1. Pick the smallest test case that proves one sub-behavior, not the whole system. One shader compiling correctly, one printed byte matching a reference — not “the driver works.”

2. Get a known-good reference for that sub-behavior before writing any agent-facing code. A hardware trace, a vendor binary’s real output, an existing filter’s byte stream. Without this, the agent has nothing to check its guesses against.

3. Hand the agent that one reference and one failing test — not the full spec. The scope of its first task is exactly as wide as the thing it needs to make pass.

4. Let it iterate against the test, then diff its output against the reference at the finest grain you can — byte by byte, or bit by bit, not “looks about right.”

5. Widen the test surface one case at a time once the narrow one passes. Never batch several new behaviors into the same round; you lose the ability to tell which change broke what.

6. Keep a standing conformance suite as the grounding check, so “it compiles” and “it runs” never get mistaken for “it’s correct” once the surface gets wide.

Test-first vs spec-first scoping: the comparison

Axis	Spec-first scoping	Test-first scoping
First artifact required	The full spec or firmware documentation	One known-good reference for one sub-behavior
First real feedback	Weeks in, once enough is implemented to run	Minutes, from the first pass/fail check
What the agent holds in context	The whole spec, mostly irrelevant to the current step	Only the slice needed to pass the next test
How a wrong guess is found	Late, and hard to localize among many guesses	Immediately, localized to the last change
Case-study result	—	GPU driver: 4 weeks, not years; printer driver: DPI bug found in one byte-diff pass

Read that table as a diagnostic, not a ranking: if your agent’s task doesn’t have a way to produce a pass/fail signal in the first hour, you’re running the left column whether you meant to or not.

What breaks if you skip test-first scoping?

Skip it and you get exactly the failure mode both projects were built to avoid: an agent that implements a large, plausible-looking surface, discovers it’s wrong only when everything finally gets tested together, and gives you no way to tell which of dozens of inferences was the actual bug. That’s expensive in wall-clock time and worse in trust — every fix becomes a fresh round of “which of these forty guesses is wrong now.”

It also wastes the one thing an LLM agent is genuinely good at here: grinding through an exhaustive, low-creativity iterate-and-diff loop without losing its place across a long session. Spec-first scoping asks the agent to do creative inference instead, which is exactly the part of the job an agent is least reliable at and a human is comparatively better at directing.

Common mistakes when scoping agents for reverse-engineering work

Handing over documentation instead of a reference. A spec tells the agent what something is supposed to do. Only a real trace or a real byte stream tells it what actually happened — and reverse engineering is precisely the domain where the two disagree.

Widening scope before the narrow case is solid. It’s tempting to let the agent run further once one piece works. Batch two or three new behaviors into the next pass and you’re back to not knowing which change caused which failure.

Treating “it compiles” as “it’s correct.” A driver, filter, or parser can build cleanly and still be wrong in exactly the field that matters. The conformance suite — CTS-style, or a byte-diff against a reference — is what actually grounds the claim.

Skipping the reference capture step to save time. Both projects spent real effort up front getting a known-good trace before writing agent-facing code. That step feels like overhead; it’s the only thing that makes every later diff meaningful.

FAQ

What is test-first scoping, and how is it different from giving an agent the full spec upfront? Test-first scoping means the agent’s very first task is to make one concrete, checkable artifact pass — a single failing test or a byte-accurate reference — rather than reading and implementing an entire specification before anything runs. Spec-first scoping hands over the whole document and hopes the agent gets every piece right before the first real feedback arrives, usually weeks in. Test-first collapses that feedback loop to minutes.

Why did the four-week GPU driver project succeed where unassisted reverse engineering usually takes years? The team scoped the agent’s work around Mesa’s own test needs instead of Apple’s full firmware documentation, so it was always working toward one pass/fail signal instead of theoretical completeness. Hardware tracing supplied a known-good reference, and the Khronos conformance suite supplied the widening surface once individual pieces worked.

Do I need an LLM agent for this, or does test-first scoping work for human engineers too? The pattern predates LLM agents — it’s the same instinct behind test-driven development and shipping a research spike as running code instead of a design doc. What an agent changes is the cost of the implement-and-check cycle: it can grind through dozens of iterative diff passes without losing its place.

What’s the minimum reference material I need before I can start test-first scoping? One known-good output to check the agent’s work against, plus one test case narrow enough to fail cleanly on the first attempt. You don’t need the full specification; you need a single ground truth and a single yes-or-no question.

Does this approach only work for graphics or driver reverse engineering? No — a GPU driver project and a macOS printer driver project used the identical structure: capture a known-good reference, diff byte by byte, widen scope only once that comparison holds. The same shape applies to any system where you can define correct as matches this trace.

How is this different from just writing more unit tests? The difference is sequencing, not coverage. Ordinary test-driven development usually starts from a spec you already understand; test-first scoping is for tasks where no usable spec exists, and the first test’s job is to force a decision on what correct even means before you write the next one.

Sources

• Building a Linux GPU Driver for the M4 Mac Mini in One Month — the primary source: the Mesa-first approach, hardware tracing via hypervisor, the 336 MB capture, and the 212 fps Minecraft result.
• Mesa 3D Graphics Library — the Gallium abstraction layer the driver is built on.
• Khronos OpenGL ES Conformance Test Suite — the standing conformance checks used as the widening test surface.
• Claude Code macOS printer driver: fixing the DPI bug HP missed — the earlier, smaller-scale case of the same reference-and-diff pattern.

The takeaway

Neither project succeeded because the model got smarter between attempts. Both succeeded because someone scoped the agent’s first task around one thing it could check, instead of one thing it had to guess. That ordering — a reference, a narrow test, a diff, then one more case — is the whole trick, and it costs nothing to apply to whatever hard, undocumented system you’re about to point an agent at next.

If you’ve read this far because you’re about to hand an agent a genuinely undocumented system, start with the printer driver case study for the smallest working version of this pattern, then writing agent tool instructions for how to encode “check before you widen” into the instructions your own agents follow. Shipping a research spike as running code covers the same test-first-over-spec-first instinct one level up, at the project-decision stage rather than inside a single session, and verifying AI agent benchmark claims covers what happens when the “test” you scoped against turns out not to be the holdout you needed. For the broader pattern of what makes a coding agent’s context effective in the first place, see the ETH Zurich study on AI coding agent context files.

Explore more: AI Coding Agents & DX