How to Verify AI Crawler IPs: 3 Lists Are Over a Year Stale

My site’s robots.txt explicitly allow-lists ClaudeBot, GPTBot, OAI-SearchBot, PerplexityBot, Google-Extended and Applebot. I wrote those lines myself, and for a while I thought of them as a policy. They aren’t — so I went looking for how you actually verify AI crawler IPs, and found the ground softer than I expected, including under the operators who are supposed to be leading here.

Crawler verification is the practice of establishing where a request really came from using evidence the sender cannot forge — its source IP, or a cryptographic signature — rather than the name it gives itself.

TL;DR

Every major AI crawler operator now publishes a JSON list of CIDR prefixes, and matching the source IP against that list — not reading the User-Agent header — is the only check that means anything today. But when I fetched all ten lists on 13 August 2026, three of them (PerplexityBot, Bingbot, Applebot) had not been regenerated in more than a year, and PerplexityBot’s covers just 8 addresses. The direction of travel is cryptographic: Web Bot Auth signs requests with Ed25519 so there is no list to go stale.

The allow-list is not a permission check

User-Agent is a client-supplied string. That is the entire story. There is no signature over it, no negotiation, no registry — the sender types it and the server believes it, or doesn’t.

A robots.txt allow-list is a routing preference, not an access control. It grants nothing, because it verifies nothing.

This matters more than it used to, because the name is now worth stealing. KnownAgents, which tracks agent traffic, is reporting an active campaign of mass vulnerability scanning that borrows AI crawler identities — ClaudeBot and ChatGPT-User among them — while probing for exactly the files you’d expect: .env, .env.production, .aws/credentials, terraform.tfstate, and, pointedly, .claude/settings.json. Their detection method is the tell: they flag traffic that “claims a recognized agent identity but fails that agent’s supported authentication method.”

The economics are obvious once you say them out loud. Sites have spent two years adding polite carve-outs for AI crawlers — softer rate limits, skipped challenges, exemptions from the bot rules. A scanner that puts ClaudeBot in its User-Agent inherits all of it for free. The allow-list didn’t create the incentive, but it did price it.

What actually verifies a crawler

The real check is the source IP, matched against a list the operator publishes and controls. Anthropic’s support documentation puts it plainly: if a crawler’s source IP is on their published list, that indicates the crawler is coming from Anthropic. Google documents two supported methods — a reverse DNS lookup that must forward-resolve back to the same IP, or a match against their published ranges.

So I fetched every list I could find and measured them. All ten returned HTTP 200 on 13 August 2026:

Two things jump out. Google regenerates daily and ships 315 prefixes with real IPv6 coverage (146 of them). And three lists — a third of the sample — haven’t been touched in over a year.

Anthropic’s entry is the interesting one. That list is new — their documentation previously read “We do not currently publish IP ranges, as we use service provider public IPs.” It now points at a live file that was regenerated yesterday. That’s the right trajectory, and it’s worth noting it only started this year.

Staleness is a failure mode, not a footnote

A stale list doesn’t fail safe. It fails quietly, and it fails in the opposite direction from the one people worry about.

If Perplexity has added a single egress IP since February 2025 — 551 days of infrastructure changes — then real PerplexityBot requests from that address fail verification. Your check says “unverified.” If you wired that to a block, you are now blocking the crawler you deliberately allow-listed, and nothing in your logs distinguishes it from the impostor you meant to stop. Both arrive as a User-Agent you trust and an IP you can’t confirm.

That is the trap. The User-Agent allow-list produces false positives — impostors waved through. A stale IP list produces false negatives — legitimate crawlers rejected. Swapping one for the other without noticing the second failure mode is how sites quietly deindex themselves from AI search.

So the correct wiring is three-tier, not binary:

• Verified — IP matched a fresh list, or reverse DNS round-tripped. Apply your allow-list.
• Unverified — no match. Rate-limit and log. Do not block on this alone.
• Hostile — unverified and requesting .env or .aws/credentials. Block on the request path, which is evidence, not on the identity claim, which isn’t.

How to verify AI crawler IPs

CIDR matching against a cached list is about fifteen lines. Fetch the list on an interval, keep it in memory, and match on request:

const LISTS = {
	claudebot: 'https://claude.com/crawling/bots.json',
	gptbot: 'https://openai.com/gptbot.json',
	'oai-searchbot': 'https://openai.com/searchbot.json',
	perplexitybot: 'https://www.perplexity.com/perplexitybot.json'
};

const toInt = (ip) => ip.split('.').reduce((acc, o) => ((acc << 8) >>> 0) + Number(o), 0) >>> 0;

function inCidr(ip, cidr) {
	const [range, bits] = cidr.split('/');
	const width = Number(bits);
	if (width === 0) return true;
	const mask = (~0 << (32 - width)) >>> 0;
	return ((toInt(ip) & mask) >>> 0) === ((toInt(range) & mask) >>> 0);
}

// prefixes: the parsed JSON, refreshed on a timer — never per-request
export function isVerified(ip, prefixes) {
	return prefixes.some((p) => p.ipv4Prefix && inCidr(ip, p.ipv4Prefix));
}

Three things that will bite you:

1. Refresh on a timer, never per-request. A fetch inside the request path turns every pageview into an outbound call and hands an attacker a trivial amplification lever. Cache for an hour; serve the last good copy if the fetch fails.
2. Handle IPv6. The snippet above is IPv4-only for brevity, and Googlebot’s list is 146 IPv6 prefixes out of 315. Dropping them means failing to verify real Googlebot traffic — the same false negative, self-inflicted.
3. Get the client IP right. Behind a proxy, req.socket.remoteAddress is your CDN. Use the trusted connecting-IP header your edge sets, and never parse X-Forwarded-For from an untrusted hop — it’s another client-supplied string, and treating it as identity reintroduces the exact bug you’re fixing.

Where this actually goes: signatures

Every problem above is a symptom of the same design flaw — identity inferred from network position, which the operator has to publish and you have to keep re-fetching. Web Bot Auth, Cloudflare’s IETF-draft standard, removes the inference.

The bot signs its request with an Ed25519 key using RFC 9421 HTTP Message Signatures. A Signature-Agent header names the domain hosting its public keys; the site fetches /.well-known/http-message-signatures-directory from that domain, verifies the signature, and checks the validity window. Cloudflare’s own write-up is blunt about why the old way is ending: IP ranges are “shared by multiple users or multiple services within the same company” and “change over time,” which makes the logic brittle by construction.

There is no list to go stale, because there is no list. An IETF working group now owns the draft, and AWS WAF shipped support. The measurement above is the argument for it: most operators can’t keep a JSON file current, and that’s the easy version of the problem.

What I changed here

Nothing in robots.txt — those allow-lists are honest documentation of intent, and this site serves identical bytes to every client by design, which is a deliberate architectural choice I’ve written about before. What changed is that I stopped describing them as a security control. They’re a preference. The agent-discovery layer that sits alongside them — llms.txt, the api-catalog, the MCP endpoint — is likewise open on purpose, and openness only stays defensible when you’re honest about what is and isn’t verified.

If you run something where crawler identity actually gates behaviour — quota, private content, a write-capable MCP tool, or an agent with network access — do the IP check, log the freshness of the list you’re checking against, and put a calendar reminder on the ones that haven’t moved since 2023.

The one-line version: treat User-Agent as a label, source IP as evidence, and a signature as proof. Most of the web is still on the first one.

Frequently asked questions

How do I verify a request is really from ClaudeBot?

Match the request’s source IP against Anthropic’s published CIDR list at claude.com/crawling/bots.json. The User-Agent header proves nothing — it is a client-supplied string, and anyone can send it. Anthropic’s documentation is explicit that an IP on that list is what indicates the crawler came from Anthropic.

Can you spoof a User-Agent like ClaudeBot or GPTBot?

Yes, trivially — it is one header in the request, set by whoever sends it. A single curl flag is enough. KnownAgents reports an active campaign doing exactly this at scale, borrowing AI crawler identities to probe for credential files like .env and .aws/credentials, on the assumption that the borrowed name buys leniency.

Do all AI crawler operators publish IP ranges?

The major ones do: Anthropic, OpenAI, Google, Perplexity, Microsoft and Apple all publish JSON files of CIDR prefixes. The catch is freshness. When I checked all ten lists on 13 August 2026, three of them — PerplexityBot, Bingbot and Applebot — had not been regenerated in over a year, while Google’s and OpenAI’s ChatGPT-User list had been refreshed within the last day.

What is Web Bot Auth?

Web Bot Auth is an IETF-draft standard, led by Cloudflare, that replaces IP guesswork with cryptography. The bot signs its request with an Ed25519 key using RFC 9421 HTTP Message Signatures, names its key directory in a Signature-Agent header, and the site fetches that directory to verify. It removes the staleness problem entirely, because there is no list to keep current.

Should I block crawlers that fail IP verification?

Not by default. A failed match means unverified, not malicious — a stale published list produces exactly the same result as a real impostor. Log and rate-limit unverified traffic, and reserve hard blocks for requests that both fail verification and probe for paths a real crawler would never request.

Does a robots.txt allow-list do any security work?

None. robots.txt is a request for voluntary compliance, addressed to a name the client chooses for itself. It is a routing preference for well-behaved crawlers, and it should be treated as documentation rather than as access control. Anything that must actually be enforced belongs at the edge, keyed on verified identity.

Sources

• Anthropic — Does Anthropic crawl the web, and how can site owners block the crawler? and the published prefix list at claude.com/crawling/bots.json
• Google Search Central — Verifying Googlebot and other Google crawlers
• Cloudflare — Forget IPs: using cryptography to verify bot and agent traffic
• KnownAgents — spoofed-agent scanning activity

List ages were measured by fetching each JSON file directly on 13 August 2026 and reading its creationTime field.