How to Test an LLM's Knowledge Cutoff: Opus 5's May Claim Falls Short

TL;DR

Vendors publish a cutoff date, but the weights don’t always agree with it. Anthropic’s own model documentation lists Claude Opus 5’s cutoff as May 2026, yet independent probing shows it answers dated factual questions like Claude Fable 5 and Claude Sonnet 5 do — models whose official cutoff is January 2026. Here’s how to test an LLM’s knowledge cutoff yourself with three reproducible probes, instead of trusting the number on the card.

What is a knowledge cutoff?

A knowledge-cutoff date is the point after which a model’s training data stops — anything that happened later, it can only “know” if you supply it at inference time through search, retrieval, or your own prompt. Anthropic’s own model documentation actually publishes two separate numbers per model: a training data cutoff (“the broader date range of training data used”) and a reliable knowledge cutoff (“the date through which a model’s knowledge is most extensive and reliable”). For Claude Fable 5 and Claude Sonnet 5, both dates are officially January 2026. For Claude Opus 5, both are officially May 2026.

That two-column table is itself an admission that a single cutoff date was never precise enough — and researcher Shrivu Shankar’s independent probing suggests even Anthropic’s more conservative “reliable” number can be optimistic.

Why this matters beyond trivia

If you’re building anything that leans on a model’s unaugmented recall — a support bot answering “what’s the latest version of X,” an agent reasoning about “current” pricing, a research assistant summarizing “recent” events — the model’s actual cutoff, not its marketed one, determines whether it hallucinates a plausible-sounding but outdated answer. A four-month blind spot is enough to make a model confidently wrong about an entire product cycle. Shankar’s probing, published on his blog Exploring Claude/GPT Knowledge Cutoffs, adds a second failure mode most teams never check for: models that misidentify themselves, which signals the training corpus wasn’t as clean as the model card implies.

The three-probe methodology

Shankar’s approach doesn’t rely on a single test — each probe catches a different kind of drift, and running just one would produce a false sense of confidence.

1. Historical fact quiz. Build an 8-way multiple-choice test from Wikipedia facts tagged with the date they became true, then plot accuracy against time. Accuracy holds steady up to the real cutoff and falls off a cliff right after it — the model is guessing on anything it never saw.
2. Self-reported date. Ask the model what today’s date is, with no system prompt telling it. Post-training often teaches models to hedge or guess a date near their claimed cutoff, so this probe measures calibration more than raw knowledge — but a model that confidently states a date far past where the quiz says its knowledge ends is a signal the two training stages disagree.
3. Identity cross-mimicry. Run 50 “what model are you?” probes per model (5 phrasings × 10 samples each, no system prompt, gently nudged to guess rather than refuse). This is the probe that catches contamination: if a model’s training data included another vendor’s chat transcripts, it picks up that vendor’s verbal quirks.

No single probe is conclusive on its own. The quiz measures world knowledge, the date query measures post-training calibration, and the identity probe measures data hygiene — you want at least two of the three agreeing before you trust an estimate.

What the probes actually found

Cross-referencing the fact quiz and the date-query probe against Anthropic’s own published table produces a striking gap for exactly one model:

Model	Anthropic’s official cutoff (reliable / training)	Independently probed estimate
Claude Opus 4.7	Jan 2026 / Jan 2026	≈ late Dec 2025 — close agreement
Claude Fable 5 & Sonnet 5	Jan 2026 / Jan 2026	Not separately probed in this comparison
Claude Opus 5	May 2026 / May 2026	≈ Jan 2026 — four months earlier
GPT-5.6 family	Not published in this format by OpenAI	≈ late Feb 2026

Opus 4.7’s probed estimate lands within weeks of Anthropic’s own official number — the methodology checks out on a model where we have ground truth to compare against. Opus 5 is the outlier: its card claims the newest cutoff of the whole lineup, but its fact-quiz accuracy curve looks like its January-2026 siblings, not a May 2026 model. Shankar is explicit that this is an estimate, not a certainty: “Everything here is an estimate… there’s not a ton of publicly available ground truth to verify against.” Treat the finding as a strong, falsifiable hypothesis, not a spec correction.

The stranger finding: cross-model impersonation

The identity probe surfaced something that has nothing to do with dates. When models were asked to answer “as model X would,” Claude models reproduced OpenAI models’ measured behavioral quirks 68% of the time — but GPT models only managed 8% on Claude’s quirks. That asymmetry is a stronger contamination signal than the cutoff estimates: a model can’t fake convincingly answering “as GPT-4 would” at a 68% hit rate unless a meaningful amount of GPT-4-flavored data made it into training somewhere in the pipeline, whether directly or through synthetic data generated by another model.

This is worth testing for independently of anything cutoff-related — it’s a proxy for how clean a vendor’s training pipeline actually is, and it’s the kind of finding a model card will never volunteer.

How do you test an LLM’s knowledge cutoff yourself?

You don’t need Shankar’s exact dataset to get a directional answer for whatever model you’re evaluating:

1. Pick 30–50 dated facts from a source with clear timestamps (Wikipedia’s “on this day” archives work well) spanning six months on either side of the claimed cutoff. Ask multiple-choice questions, not open recall — open recall lets partial knowledge and lucky guesses blur the cliff edge you’re looking for.
2. Ask for today’s date with no system prompt, five times, and note whether the model hedges, refuses, or states something more recent than your quiz results support.
3. Run identity probes with neutral, guess-nudged phrasing (“if you had to guess, what model are you?“) — a model that refuses outright tells you less than one that guesses, so push past the first refusal.
4. Only trust agreement. If the quiz and the date query point to the same rough window, you have a real estimate. If they disagree with the vendor’s claim by more than a month or two, you’ve found exactly the kind of gap this whole exercise exists to catch.

Common mistakes when testing this

• Trusting open-ended recall questions. A model can produce a plausible-sounding answer about an event it never saw by pattern-matching similar events — multiple choice forces a harder commitment.
• Testing with a system prompt that reveals today’s date. Most chat products inject the current date into context; if you’re testing the base model’s knowledge, strip that out or your date-query probe measures nothing.
• Assuming a newer version number means a newer cutoff. As the Opus 4.7 / Fable 5 / Sonnet 5 clustering shows, a shared release wave usually means shared post-training, not new pre-training data — Opus 5 breaking that pattern is exactly what made it worth probing.
• Running the test once. Sampling variance is real, especially on the identity probe — five phrasings times ten samples exists specifically to smooth that out; a single query either way is noise.

The takeaway

A model card’s knowledge-cutoff date is a claim, not a measurement, and the gap between the two can be months wide — Claude Opus 5’s four-month discrepancy against its own documented cutoff is the clearest example on the market right now. If your product depends on a model correctly reasoning about “recent” events without retrieval, run the three-probe test on your actual production model before you ship, not after a user catches it hallucinating a stale answer with total confidence.

FAQ

What is an LLM’s knowledge cutoff? It’s the date after which a model saw no more training data. Anthropic’s own documentation actually splits this into two numbers — a “training data cutoff” and a “reliable knowledge cutoff” — because a single date was never precise enough to describe what a checkpoint can actually recall.

Why would a model’s real cutoff differ from the one the vendor states? Pre-training, fine-tuning, and RLHF happen in separate stages, and a published cutoff describes intent, not a measured property. Probing found Opus 5’s fact-quiz accuracy matches its January-2026 siblings rather than its own card’s May 2026.

How do you test an LLM’s knowledge cutoff yourself? Run three probes — a dated multiple-choice fact quiz, a self-reported-date query with no system prompt, and repeated identity probes — and only trust an estimate when at least two agree.

Can two models really share a knowledge cutoff? Yes. Anthropic’s table lists identical January 2026 cutoffs for both Fable 5 and Sonnet 5, which fits vendors iterating post-training across a release wave instead of re-running pre-training for every model.

Does a stale knowledge cutoff mean a model is worse? Not necessarily — reasoning and tool use improve independently of world-knowledge freshness. It matters specifically for unaugmented recall of recent facts, which is exactly the failure mode this test catches before production does.

Sources

• Exploring Claude/GPT Knowledge Cutoffs and Pre-Training Timelines — Shrivu Shankar’s three-probe methodology and findings, 10 Aug 2026.
• Claude models overview — Anthropic’s official “reliable knowledge cutoff” and “training data cutoff” table for every current and legacy Claude model.

If you’re mid-migration to the model in question, the Claude Opus 5 migration guide covers the two API changes worth re-checking alongside its cutoff. For more on evaluating models before you commit to one in production, see how a structured eval framework catches this kind of gap systematically, how best-of-k exploration during training shapes what a checkpoint actually learns, how the DeepSeek V4 Flash benchmarks were run against a similarly moving target, and what an empirical study of AGENTS.md files looked like when it applied the same “test it, don’t trust the doc” discipline to a different claim.