How to Stop AI Scrapers Overloading Your Server: the 20% CPU Toll

TL;DR How to stop AI scrapers overloading your server, in one line: proof-of-work, not blocklists — User-Agent and IP bans on git.kernel.org got circumvented within weeks, so the project now spends roughly 14 to 16 of its 90 CPU cores, about a fifth of total capacity, rendering commit pages for bots that never return. Its fix, a challenge system called Anubis, forces each visitor’s browser to solve a small cryptographic puzzle before loading a page — trivial for a human, expensive at scraper scale — and it already had to raise the difficulty once as scrapers caught back up. The durable fix isn’t the puzzle itself; it’s shrinking how much expensive, crawlable surface exists to hit in the first place.

If your server has ever had a CPU graph that never comes down at 3am with no matching spike in real users, this is why. Konstantin Ryabitsev, who runs kernel.org’s git infrastructure, wrote up the numbers in detail, and they’re a clean case study in exactly how this fight escalates and where it actually stops.

What is a proof-of-work challenge?

A proof-of-work challenge is a small computational puzzle — usually “find an input whose SHA-256 hash starts with N zero bits” — that a visitor’s browser must solve before the server hands over a page. The puzzle is deliberately asymmetric: verifying a solution takes the server microseconds, but finding one takes the client real, non-negotiable CPU time that scales with the difficulty you set. A human loading one page pays that cost once and doesn’t notice it. A scraper trying to render every commit, diff, and file-blame page across nearly a million commits pays it millions of times over, which is exactly the leverage a rate limit or IP ban doesn’t give you.

Anubis, the tool kernel.org deployed, sits in front of cgit and issues exactly this kind of challenge to anonymous traffic. It’s open source and increasingly common in front of forges, wikis, and docs sites that got hit the same way — the project is on GitHub if you want to see the actual challenge implementation.

Why git.kernel.org needed one at all

The numbers here are what make this worth taking seriously instead of filing under “annoying logs.” Across five geo-distributed nodes totaling 90 CPU cores, git.kernel.org was seeing about 6 million requests a day hitting effectively random commit URLs. At any given moment, 14 to 16 of those 90 cores — roughly a fifth of total fleet capacity — were doing nothing but rendering git commits as HTML for scrapers that would never open a second session. Ryabitsev’s own estimate, made under generous assumptions, put legitimate human traffic at around 2% of the total.

The economics only make sense once you see why scrapers bother at all: pre-2020 kernel history is some of the cleanest, most abundant, and most verifiably human-written code in existence, which makes it valuable as guaranteed-uncontaminated training data. That’s a strong enough incentive that scrapers will pay real infrastructure cost to collect it, even when the data is already available as a git clone that would cost the scraper operator less bandwidth and cost kernel.org nothing in rendering CPU.

How the mitigation arms race actually played out

Every cheap defense worked for a while and then stopped working, in a pattern that repeats across almost every site fighting this problem:

1. User-Agent string blocking — worked immediately, until scrapers started sending headers indistinguishable from a real browser.
2. IP-based bans via fail2ban — worked until the traffic moved to distributed residential and mobile proxies, where a single IP makes four or five requests and is never seen again. There’s no point banning an address that won’t come back.
3. ASN-level blocking — held slightly longer, but started catching legitimate automated tools sharing hosting ranges with abusive traffic.
4. Proof-of-work at low difficulty (Anubis, 4 leading zero bits) — stopped the unsophisticated bots outright; even mobile devices solved it without anyone noticing.
5. Proof-of-work at higher difficulty (5 leading zero bits) — bought “a few more months of peace,” in Ryabitsev’s words, at the cost of phones becoming noticeably warm while solving it. Scrapers resumed within months, now solving difficulty-5 challenges as a matter of course.

Each rung of that ladder raised the attacker’s cost without changing the fundamental shape of the problem: as long as there’s a URL to hit, something will eventually be willing to pay whatever the current toll is to hit it.

How to stop AI scrapers overloading your server without blocking humans

If your own traffic graphs look like kernel.org’s, the deployment order that avoided collateral damage there is worth copying directly:

1. Measure which routes are actually expensive first. Kernel.org’s cost wasn’t uniform — it was concentrated in per-commit and per-diff rendering, not static pages. Instrument origin CPU by path before you touch anything.

2. Take the free wins, but don’t trust them to last. User-Agent and known-bad IP/ASN blocking still catch the least sophisticated traffic today. Deploy them, and plan for them to degrade within weeks.

3. Allowlist known-good crawlers before turning on proof-of-work. Match published CIDR ranges for search engines the same way you’d verify any other bot, so you don’t accidentally puzzle-gate the traffic you actually want indexing you.

4. Put the challenge only in front of the expensive paths, starting at the lowest difficulty that meaningfully deters automated traffic. Kernel.org’s difficulty-4 tier was invisible to real visitors and still stopped most bots cold.

5. Raise difficulty reactively, not preemptively, and watch for real-user cost (battery, perceptible delay) before you do — difficulty 5 bought time but came with a cost real visitors could feel.

6. Shrink the crawlable surface as the actual long-term fix. Kernel.org’s own move was cutting the number of rendering options and URL variants per commit, because reducing what exists to be scraped beats raising the price of scraping it, indefinitely.

What breaks when scrapers start solving your challenge

The uncomfortable finding in Ryabitsev’s writeup is that even a working defense has a half-life. Of the 6 million daily requests, roughly two-thirds get blocked at the perimeter before any proof-of-work is even asked for — but the remaining third still gets through, and scrapers are increasingly willing to burn the CPU to solve difficulty-5 challenges as a routine cost of doing business, not an obstacle. The proxy infrastructure behind this has also industrialized: some of this traffic is now routed through compromised residential IoT devices — the post specifically calls out smart TVs — monetized as SDK-based proxy networks, which is what makes IP-based blocking permanently behind the curve.

That’s the actual argument for treating the puzzle as a delay tactic rather than a solution: it buys months, not permanence, and every difficulty increase you add is a cost your real users partly absorb too.

Bot mitigation techniques compared

Technique	Cost to bypass it	Cost to real users	How long it held at kernel.org
User-Agent string blocking	Trivial — fake the header	None	Days to weeks
IP bans (fail2ban)	Low — rotate residential proxies	None, unless false-positived	Weeks
ASN-level blocking	Medium — avoid flagged ranges	Risk of blocking legitimate automation	Weeks to months
Proof-of-work, difficulty 4	Medium — added CPU per request	Imperceptible, even on phones	A few months
Proof-of-work, difficulty 5	High — noticeable CPU/heat cost	Noticeable delay, phone warms up	“A few more months,” per Ryabitsev
Reducing crawlable surface	Structural — no URL, no target	Fewer convenience features for anonymous users	Ongoing; the current approach

The pattern in that table is the whole lesson: every row above the last one is a toll increase, and every toll increase gets paid eventually. Only the last row changes the game instead of the price.

Frequently asked questions

What is a proof-of-work challenge for bot defense?

It’s a small cryptographic puzzle a visitor’s browser must solve before the server returns a page, typically finding an input whose hash has a required number of leading zero bits. A human’s browser solves it in a fraction of a second and never notices; a scraper hitting millions of pages pays that cost on every single request, which is the point. Anubis, the tool git.kernel.org uses, implements exactly this pattern in front of cgit.

Why did User-Agent and IP blocking stop working against AI scrapers?

Because both are cheap for an attacker to fake or route around. Scrapers started sending legitimate-looking User-Agent headers once naive string matching became common, and when kernel.org moved to IP-based bans via fail2ban, the traffic simply shifted to distributed residential and mobile proxies — individual IPs now make four or five requests and disappear, so there’s rarely a repeat offender worth banning. ASN-level blocking held slightly longer but caught legitimate automated checkers in the process.

How much does a proof-of-work challenge slow down a real visitor?

At the difficulty git.kernel.org first deployed (four leading zero bits), the delay was imperceptible even on phones. Raising it to five bits bought a few more months of relief but made mobile devices noticeably warm while solving it — still under a second on most hardware, but a real, measurable cost that has to be weighed against how much it’s actually still deterring scrapers.

Does Anubis block search engine crawlers too?

It can, if configured to challenge everyone indiscriminately, which is why most deployments allowlist known-good crawler ranges (the same published CIDR blocks you’d use to verify Googlebot or Bingbot) before turning proof-of-work on for everyone else. The failure mode to avoid is treating all bots as equivalent — a search crawler indexing your public docs is not the same threat as a scraper harvesting your entire commit history for training data.

What’s the real fix if proof-of-work eventually gets circumvented?

Shrinking the attack surface, not raising the difficulty forever. Kernel.org’s own conclusion was to turn off features that generate what it called “1.2 metric bajillion” crawlable URLs per fork — every commit, diff, and rendering option was a separate indexable page — because no amount of per-request friction beats simply not exposing the expensive path at all.

Is this only a problem for huge projects like the Linux kernel?

No — it’s a scale problem, not a fame problem. Any git host, forum, or docs site that renders content dynamically per URL is exposed to the same math: a small number of distinct real visitors versus an unbounded number of URLs a crawler can enumerate. Smaller sites just hit the CPU ceiling later, not never.

Sources

• Konstantin Ryabitsev, “Creepy crawlies” — the original numbers and timeline from git.kernel.org’s infrastructure team.
• Simon Willison, commentary on the same post, noting the comparable overhead pattern on Datasette-hosted sites.
• Anubis (TecharoHQ) — the open-source proof-of-work challenge system referenced throughout.

If you’re already fighting scraper traffic that ignores robots.txt, the next two problems you’ll hit are proving a request really came from the crawler it claims to be — see verifying AI crawler IPs instead of trusting the User-Agent — and keeping your training-data opt-out in sync as enforcement rules change, covered in syncing robots.txt without losing search visibility. The same asymmetry — cheap for an attacker to probe, expensive for you to serve — shows up again once content is inside your walls; Anthropic’s approach to detecting distillation attacks is the same arms race one layer up the stack. And if you’re the one absorbing traffic spikes at the origin rather than the edge, the layered cache architecture that let one CMS absorb a 28,000 RPS DDoS is a useful comparison for how much perimeter defense actually costs to build.

None of this makes the problem go away — Ryabitsev is explicit that kernel.org still promises all its data to anyone who asks, which means the fix is friction for anonymous bulk access, not a wall. If you’re seeing the same CPU graph, start with the free blocks, add proof-of-work only in front of what’s actually expensive, and treat every difficulty increase as bought time, not a finish line.