← All media signals
Infrastructure

The Consent Layer

Websites can tell AI crawlers how they may use a page, and an EU code has providers mark AI outputs, but those crawler rules are requests, not locks.

What is it?

Machine-readable permission rides on two tracks: site-level rules telling AI crawlers what they may do with a page, and EU-coded marks on AI outputs. The formats differ, and none of the crawler rules enforces itself. Whether any model actually obeys them stays open.

Why now?

Generative tools make content trivially reproducible, so publishers want a say before a crawler takes a page. Cloudflare split crawlers into search, agent and training bots, and an EU code has around 190 signatories marking AI outputs. Whether that check ever becomes obligatory is unresolved.

Who’s building it?

Cloudflare and the European Commission set the formats; publishers set the rules, and AI crawlers decide whether to honour them.

European CommissionCloudflarePublishers · platformsAI crawlers · model buildersRegulators · courts
The full read
Why now, who is building it, where the market goes, and what stays unsettled.
Get weekly →
01 · The market
Who’s in it
Builders — standards groups and platforms building the machine-readable signal itself.
Buyers — publishers and site owners who want their crawler rules respected.
Gatekeepers — AI companies and crawlers deciding whether to honour the signal.
What’s at stake
Whoever controls the verification checkpoint decides what gets trained on — the leverage sits above the creators it protects.
Several standards compete, but consolidation around one seems likely.
Reputational cost falls on whoever chooses to ignore it.
What’s unsettled
Whether any signal becomes binding, or stays a polite suggestion.
Which single standard wins out, or whether several simply coexist.
Whether creators who set the rule actually gain leverage, or platforms absorb it anyway.
02 · How the demand matureda progression, not a timeline
Labelling
AI providers mark outputs in machine-readable form under an EU code.
Enforcement
Robots.txt states a crawler preference but does not enforce it.
Consolidation
A dominant standard emerges as courts and platforms start treating one label as the real one.
Liability
Ignoring an embedded signal turns from a reputational risk into a legal one.
03 · Where the market goesthe forks, not a prediction
→Converge or fragment — either the standards merge into one enforceable rule, or the web ends up carrying several competing, semi-compatible signals at once.
→Regulators could force the signal to matter — or leave enforcement entirely to voluntary platform goodwill.
→Model builders may start honouring the signal by default, or keep training on whatever they can reach.
→The checkpoint’s value could sit with the standard, or with the verifier.
The demand question underneath it all: will model builders treat the signal as binding, or as noise they can route around indefinitely.
04 · The Verification Stack deskadjacent markets, moving now
The Trust Layer
Watermarking AI output at platform scale.
SynthID has watermarked over 100 billion images and videos.
Mashable
The Provenance Web
Provenance labels spreading across major platforms at scale.
TikTok labeled over 3 billion AI-generated posts via C2PA.
C2PA
Post-Password
Verification moves into machine-readable form at scale.
About 190 organisations signed the EU code to mark AI outputs.
European Commission
05 · The Wire
01
Cloudflare lets sites set separate rules for search, agent and training bots.
Help Net Security ↗
02
EU code asks AI providers to mark outputs in machine-readable form.
European Commission ↗
The panel on this signal
They agreed on
–Permission is migrating from legal terms into machine-readable signals, though none of them yet binds a crawler.
–Leverage sits with whoever actually runs the checkpoint itself.
–Publishers and platforms are setting the norms well ahead of any actual law.
The privacy counsel disagreed on
–A machine-readable preference signal carries no binding force absent a legal duty to honour it; without enforcement mechanisms or liability attached to non-compliance, the marker functions as a label rather than an enforceable right.
Know more about the method →
✦How this signal was built
01 · Panel
A Delphi panel of 4 AI analysts, each reasoning independently.
→
02 · Debate
They challenge and revise each other’s reads over two rounds.
→
03 · Evidence
Every claim is tied to a cited, linked source on The Wire.
→
04 · Synthesis
A moderator writes it into one read, disagreement and all.