Build on-device AI without breaking E2EE: the metrics leak first

TL;DR

Building on-device AI without breaking E2EE is not mainly a modelling problem — keeping inference local is the easy half. WhatsApp’s Scam Alert, detailed by Meta on August 12, 2026, spends most of its architecture on the two things that leak after inference: the metrics you send home to improve the model, and the absence of any way for an outsider to check that you did what you claimed. Its answer is two trusted execution environments, k-anonymity plus differential-privacy noise on every released number, and a model hash published to a Cloudflare-operated append-only ledger before the model is served to anyone.

What is Scam Alert, and what does it actually guarantee?

Scam Alert is, in Meta’s words, “an optional feature that runs an on-device machine learning model to alert a user about potential scam messages.” It looks at incoming messages from people who are not in your contacts, classifies them on “conversational structure and linguistic signals,” and if the model fires, shows a warning inside the chat. The sender never sees it. The feature is in Limited Beta as of the engineering write-up.

The load-bearing sentence is this one: “All inference happens on-device and no message content leaves the user device for classification.” That is the guarantee everyone assumes when they hear “on-device AI.” It is also, by itself, worth much less than people think.

Note two design choices that already look unusual. The model is downloaded from a CDN, not hardcoded into the app — which decouples model updates from app releases and, more importantly, makes the model a distributable artifact that can be independently pinned and checked. And the user’s available actions are block, report, or mark-as-trusted, with “WhatsApp is unable to initiate sharing of any user data without the user’s action.” Reporting is a user gesture, never an automatic upload.

The three guarantees behind on-device AI without breaking E2EE

Meta frames the design as three commitments, and they map cleanly onto the three ways this class of feature normally fails:

Guarantee	The failure it prevents	Mechanism
No message content leaves the device	Classification becomes a plaintext upload	Local inference only; reporting is user-initiated
No targeted model delivery	You get a special classifier and can’t tell	Every version’s SHA-256 published to a third-party append-only ledger before it is served
Confidential analytics	Telemetry re-identifies what inference protected	Two TEEs, k-anonymity thresholds, differential-privacy noise

Most teams ship row one and stop. Rows two and three are where the interesting engineering is, because they are the rows that answer “why should I believe you?” rather than “what does the code do?”

Your metrics pipeline is the part that leaks

Here is the trap. You have done the hard privacy work — the model runs locally, nothing is uploaded. Now you need to know whether the thing works: how often does it fire, how often do users block versus dismiss, is the false-positive rate drifting? So you reach for the analytics SDK already in the app and emit an event.

That event has a device identifier, an IP address, and a timestamp. It says a scam warning fired. You have just built a server-side record that a specific user received a message your classifier considered a scam, at a specific moment — which is a meaningful slice of exactly the metadata E2EE exists to withhold. The inference never touched your servers and it did not need to.

Scam Alert’s answer is a four-hop path where each hop removes a different thing:

Walking it in order: the device authenticates through an anonymous credentials service so it can prove it is a legitimate client without revealing which account it is. The request then travels via an OHTTP relay, which strips the source IP — the relay sees the address but not the payload, the recipient sees the payload but not the address. It arrives at a stateless orchestrator hosted on a TEE, which validates the privacy configuration and batches metrics. The batch goes to the aggregator TEE, which merges contributions into running histograms, “enforces k-anonymity thresholds to suppress results with too few contributors,” and applies differential-privacy noise. Only then does a number become visible to Meta, and only as what the write-up calls “approximate, aggregate counts.”

The pattern generalizes past this one feature: the privacy property of a machine-learning feature is set by its weakest data path, not by where inference runs. Anyone who has sandboxed an AI agent’s network access has met the same lesson from the other side — the model layer was never the layer that leaked.

Two details are worth stealing. The aggregator’s partial state is held in encrypted recovery checkpoints whose keys only attesting TEEs hold, so a crash mid-aggregation cannot spill a half-finished pile of individual contributions. And this is not bespoke code: it is the PAPAYA federated analytics stack, published at USENIX NSDI ‘25, reused. Peer-reviewed plumbing you can point at beats a novel design nobody has read.

Verifiability is a separate engineering problem

Everything above is a claim about code you cannot see. The genuinely novel half of this design is making those claims checkable by someone who does not trust Meta.

It works in two directions from a single root of trust.

Downward, for the model. “Every model version — including its SHA-256 hash — is published on a third-party append-only transparency ledger before it is served to anyone.” The ledger is operated by Cloudflare, and the write-up is explicit that Meta does not hold the signing key. Cloudflare has audited WhatsApp’s Auditable Key Directory since 2024, so this reuses a trust relationship that already exists rather than inventing one. Entries are publicly resolvable under akd-auditor.cloudflare.com by namespace and epoch. The property that buys: nobody can be quietly served a bespoke classifier, because the hash of what you received either appears in an append-only log or it does not.

Upward, for the pipeline. The client opens a Remote Attestation + TLS (RA-TLS) session with the orchestrator TEE. The attestation quote carries measurements of the code actually running there, and the client “cross-checks [them] against a third-party transparency ledger to ensure it is connecting only to code that satisfies our verifiable transparency guarantee.” If the measurements are not in the ledger, the client does not send its metrics. That is what turns “we use TEEs” from a slide into a runtime check.

Weights are published too, so researchers can analyse what the classifier actually keys on, and the user gets an on-device log of what was flagged at Account → Request Info → Scam Alert Activity. The verifiable-control pattern behind Cloudflare’s WriteGuard is the same instinct at a smaller scale: a control an outside party can audit outranks a control you merely assert.

Where this design usually gets copied wrong

Four failure modes, in the order teams hit them:

1. Local inference, centralized telemetry. The single most common one. Inference is local, metrics are not, and the metrics carry identity. The fix is not “anonymize later” — noise applied to a table that already holds identifiable rows is a reporting control, not a privacy guarantee.

2. DP noise without a k-anonymity floor. Differential privacy bounds the influence of any one contributor on a published aggregate. It does not stop a bucket with three contributors from being informative about those three. Scam Alert pairs the two deliberately.

3. TEEs without attestation. Running the aggregator in a confidential VM changes nothing if the client will happily hand its data to whatever answers the socket. RA-TLS against a published measurement is what makes the TEE load-bearing.

4. A ledger you control. An append-only log you can sign and rewind is a database with extra steps. The value comes precisely from the signing key sitting outside your organisation.

There is also a scope question worth being honest about: the classifier still reads your messages — on your hardware, at your option, but it reads them. Scam Alert’s answer is that the feature is optional, scoped to non-contacts, and produces an on-device log; the last-5-messages sharing that improves the model is a separate, explicit opt-in attached to marking a chat trusted. “The user chose it and can inspect it” is a different guarantee from “nobody can see it,” and conflating them is how privacy features lose trust.

What to copy if you are building this

You will not have a Cloudflare-audited ledger on day one. You can still take the shape:

• Ship the model as a versioned artifact with a published hash, even if the “ledger” is a signed manifest in a public repo. It makes targeted delivery detectable, which is the whole point.

• Default telemetry to nothing and add counters individually, each one justified in aggregate form. This is the decision that is expensive to reverse — every other layer here can be added later.

• Put a k-anonymity floor in the query path, not in the dashboard. Suppression an analyst can toggle off is not suppression.

• Strip identity at a hop you do not own. An OHTTP-style relay is the cheapest available version of “we cannot correlate this even if we wanted to.”

• Write down what you are not guaranteeing. The features that survive scrutiny are the ones whose limits were stated by the vendor first.

Platform-side scam detection — the kind that reads content server-side and bans networks of accounts, as in OpenAI’s abuse-detection work — catches things a single device never can, because it sees the graph. On-device detection sees one conversation and nothing else. That is a genuine capability cost, paid deliberately in exchange for the encryption guarantee, and the architecture above is what makes the trade legible instead of a marketing line.

The takeaway

The interesting part of Scam Alert is not that a small classifier runs on a phone; models have run locally on consumer hardware for years. It is that Meta treated “why should anyone believe the privacy claim?” as an engineering requirement with a concrete answer — a hash in someone else’s append-only log, an attestation quote checked at runtime, and a metrics path where no single hop holds both identity and content. If you are building a privacy-preserving ML feature, that is the part to copy, and the metrics pipeline is where you should start.

FAQ

Does on-device inference alone preserve end-to-end encryption? Only for message content, and only while nothing else about the classification leaves the device. A feature is more than its inference call — it ships a model, reports whether that model fired, and lets the user act on the result. Each is a channel that can re-identify the user.

What is confidential federated analytics? Computing aggregate statistics across a device fleet without any single party seeing an individual contribution. Devices encrypt metrics to a TEE rather than to the operator; only aggregates that clear a k-anonymity threshold and carry differential-privacy noise are released. Meta’s implementation is the PAPAYA stack from USENIX NSDI ‘25.

Why publish the model hash before the model ships? To make targeted model delivery detectable. Publishing each version’s SHA-256 to a third-party append-only ledger before it is served lets a client check its copy against a record the vendor cannot retroactively edit.

What does RA-TLS actually verify? That the encrypted channel terminates in code someone committed to in advance. The attestation quote carries measurements of the orchestrator TEE, which the client cross-checks against the transparency ledger before sending anything.

Is differential privacy enough on its own? No. DP bounds one contributor’s influence on a published aggregate; it says nothing about who saw the raw contributions. That is why the noise is applied inside the aggregator TEE and paired with a k-anonymity threshold.

What is the smallest version I can build? Local inference, telemetry defaulted to nothing, a relay that strips the IP, and a k-anonymity floor in the query path. Attestation is an upgrade you can add; an identifiable metrics pipeline is a rewrite.

Sources

• Meta Engineering — How We’re Building Scam Alert on WhatsApp With End-to-End Encryption and Verifiability Guarantees (August 12, 2026)
• Srinivas et al. — PAPAYA Federated Analytics Stack: Engineering Privacy, Scalability and Practicality, USENIX NSDI ‘25
• Cloudflare — Auditing key transparency for end-to-end encrypted messages