Cloudflare's AI Crawler Defaults Took Effect on September 15, 2026 — Here's What Actually Changed

The short version: Cloudflare's new AI crawler defaults took effect on September 15, 2026, but not quite as announced in July. Search crawlers stay allowed. On new ad-supported sites, AI training is now set to a new "Disallow AI Training" preference, and AI agents are blocked on pages with ads. The biggest change from the July plan concerns mixed-use crawlers such as Googlebot and Applebot. They are no longer blocked when a site opts out of training: Cloudflare labels them "Accountable" and lets them keep crawling for search, while training-only crawlers are blocked. Sites that used the old "Block AI bots" switch were migrated to the new settings automatically.

This guide covers what took effect on September 15, what Cloudflare announced in July, who is affected, and what to do now, whether you run crawlers or run a website.

Key dates at a glance

• July 1, 2026 — Cloudflare announces the new crawler taxonomy and defaults (its second "Content Independence Day"; the first, July 1, 2025, introduced the original one-click AI-bot block).
• July 1 → September 14, 2026 — the new AI traffic controls were live in every Cloudflare dashboard, so customers could set their own preferences before the defaults kicked in.
• September 15, 2026 — the defaults take effect with a new "Disallow AI Training" setting. New ad-supported sites allow search, disallow AI training and block AI agents on pages with ads, and sites on the old "Block AI bots" setting are migrated to the same combination.
• Early 2027 — Microsoft's target for honoring a robots.txt "no training" preference for Bingbot.

What actually happened on September 15, 2026

On September 15, 2026, Cloudflare switched on its new AI crawler defaults and changed the most controversial part of its July plan at the same time. Instead of blocking mixed-use crawlers such as Googlebot wherever a site blocks AI training, it introduced a "Disallow AI Training" setting. The setting writes a no-training rule into the site's robots.txt, blocks training-only crawlers, and lets "Accountable" mixed-use crawlers keep crawling for search. The old one-click "Block AI bots" toggle was replaced by three separate controls for search, AI training and AI agents. The managed robots.txt feature was replaced by "Bot Preference Sync".

What "Accountable" means

Cloudflare labels a crawler Accountable when its operator offers, or has committed to, four things:

• a way to opt out of AI training through robots.txt or a similar standard;
• a way to opt out of AI summaries;
• URL-level visibility into which pages were used for training and how content appeared in search;
• an assurance that opting out of training won't affect traditional search results.

According to Cloudflare, Googlebot and Applebot meet the criteria today, through Google-Extended and Applebot-Extended. Microsoft is still building robots.txt "no training" support for Bingbot, targeted for early 2027. Amazon, Anthropic, Meta and OpenAI qualify because they already run separate search and training crawlers.

The defaults as launched

Existing sites that had set Training to "Block" or "Block on pages with ads" in the granular controls were moved to Disallow AI Training as well. Choosing Block for training is still possible, but it now stops Googlebot, Applebot and Bingbot entirely, which removes the site from their search results.

Cloudflare's own numbers explain the change of course. 17% of sites on its network restrict AI training in some way, while fewer than 1% block search crawlers, and mixed-use crawlers make up 36.6% of verified crawler traffic. CEO Matthew Prince described the result as "preserving the openness that makes search valuable while giving the people and businesses behind the web meaningful control over how their work is used."

What Cloudflare announced on July 1, 2026

Cloudflare's announcement is a response to a milestone the company itself measured: bots now generate the majority of web traffic, and AI crawlers are the fastest-growing slice of it. CEO Matthew Prince framed the change bluntly: "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge." Cloudflare also noted that more than half of all AI crawler traffic is wasted effort — re-fetching pages that haven't changed since the last visit.

The core of the announcement is a shift from a single on/off switch ("block AI bots") to a purpose-based system. Cloudflare no longer asks whether a visitor is a bot — it asks what the bot is for, and applies different default rules to each purpose. Site owners get granular controls; crawler operators get an ultimatum: separate your crawlers by purpose and declare that purpose honestly, or get treated as the most-restricted category you blend in.

The new crawler taxonomy: Search, Agent, Training

Cloudflare now sorts AI crawlers into three primary categories, each with its own default treatment:

Why "ad-supported pages"? Cloudflare's reasoning: "An ad is a signal that a website owner meant for a person to land there and see it." If a page is monetized by human attention, a bot that consumes the content without delivering a human is, by default, unwelcome.

The taxonomy goes further than the big three. Cloudflare now also tracks Transact, Data Collection, Security Testing, SEO, Ads Verification, Social/Link Preview, Feed Fetching, and Monitoring & Operations as distinct bot purposes — meaning legitimate price monitoring, ad verification, and uptime tooling now have named lanes of their own rather than being lumped in with AI training.

The mixed-use problem: the July plan for Googlebot

Update, September 15, 2026: this part of the plan changed before launch. Under the new Disallow AI Training setting, Googlebot and Applebot stay allowed for search and only training is refused; see what actually happened on September 15. The original July plan is kept below for reference.

The most consequential detail is the treatment of mixed-use crawlers — bots that perform search indexing and model training with the same user agent. Googlebot, Applebot, and Bingbot are Cloudflare's own examples. Under the July plan, a multi-purpose crawler "will be blocked by customers who have selected to block Training" — the crawler inherits the treatment of its most-restricted purpose. Google disputes the framing, pointing to its Google-Extended token that lets sites opt out of training use without affecting Search rankings, but Cloudflare's system puts the burden on the crawler operator to run separate, honestly-labeled crawlers per purpose.

Who is affected

In July, Cloudflare said the new defaults would apply to new domains, new sites created by existing customers, and existing free-tier zones. As launched on September 15, Cloudflare's documentation describes defaults for new sites, split by whether they carry ads, and an automatic migration for existing sites that already blocked AI bots, which moved to Disallow AI Training. Other existing customers keep their settings and are asked to review and confirm their preferences as Bot Preference Sync replaces the managed robots.txt.

The direction of travel is unchanged. Cloudflare fronts roughly a fifth of the web, and every new ad-supported site on it now starts out refusing AI training and blocking AI agents on ad pages, while staying open to search.

Content Signals: the robots.txt extension with teeth

Update, September 15, 2026: Cloudflare's managed robots.txt has been replaced by Bot Preference Sync, which writes your dashboard settings into robots.txt as standard User-agent and Disallow rules for each crawler. Check your own robots.txt to see exactly what is now published for your site.

Alongside the blocking defaults, Cloudflare extended its managed robots.txt with a machine-readable preference line:

Content-Signal: search=yes,ai-train=no,use=reference

The new use field declares one of three permitted content-use levels:

• immediate — a bot may interact with the content but store and reuse nothing;
• reference (the default) — a bot may index, excerpt, and link back;
• full — a bot may summarize and reproduce the content.

Two things make this more than a polite suggestion. First, Cloudflare ties it to enforcement: a bot that reproduces content in full cannot achieve Verified status, and unverified bots fall into the default-blocked pool. Second, the legal context has shifted — under the EU AI Act, whose general-purpose AI obligations become enforceable on August 2, 2026, machine-readable opt-outs carry real legal weight for anyone training models on scraped data. A robots.txt line is no longer just etiquette; ignoring it is becoming a documented, timestamped liability. (For the broader legal picture, see our guide to proxy and web data compliance.)

Verified bots and Web Bot Auth: identity over IP

The verification system changed in a subtle but important way. Previously, a Verified bot was allowed by default — verification was the permission. Now verification and permission are decoupled: "Verified" means a bot is allowable within its relevant category. A verified training crawler is still blocked on a page where training is blocked. To earn and keep Verified status, operators must demonstrate honest self-representation and no abuse of access.

Cloudflare also proposed a mechanism for transitive trust — carrying a bot's identity and declared purpose through intermediaries like proxy layers and agent infrastructure, using the standard Forwarded header from RFC 7239:

Forwarded: for="openai";use="reference"

This sits on top of Web Bot Auth, the cryptographic bot-identity scheme (HTTP Message Signatures) that Cloudflare moved into production earlier this year. The strategic signal for the data industry is hard to miss: the web's gatekeepers are migrating from IP reputation ("where does this request come from?") to declared, verifiable identity ("who is this and what will they do with the content?"). IP quality still decides whether undeclared traffic survives first contact — but for declared bots, identity is becoming the passport.

Pay Per Crawl becomes Pay Per Use

Cloudflare's Pay Per Crawl marketplace — which lets sites charge crawlers per request — is evolving into Pay Per Use, which lets publishers charge when their content actually creates value downstream, not just when it is fetched. Launch partners are Ceramic.ai and You.com: publishers earn when their content surfaces in Ceramic's AI search results or when You.com accesses premium content. For data teams, the takeaway is that "blocked" and "free" are no longer the only two states a page can be in — licensed access at a price is becoming a first-class option, and it will increasingly be the compliant path to content that the defaults now wall off.

What data teams should do now

If you operate crawlers, agents, or any automated data collection, here is a concrete audit:

1. Inventory your exposure. Identify which of your target domains sit behind Cloudflare and how much of your collection volume they represent. That is the traffic the September 15 defaults can touch.
2. Classify your own traffic against the taxonomy. Is each workload search, agent, training, data collection, SEO, ads verification, or monitoring? If one crawler does several of these under one user agent, split it into separate, honestly labeled crawlers per purpose. That is exactly what earns Cloudflare's "Accountable" label.
3. Check whether your targets are ad-supported. The default block applies to pages carrying ads. Content APIs, docs, and ad-free pages are outside the default (though owners can still block manually).
4. Pick a lane: declared or undeclared. If your use case can live with the rules — referral-friendly indexing, agent traffic acting for a real user — apply for Verified status, implement Web Bot Auth signatures, and honor Content Signals. If you stay undeclared, understand that you are choosing the adversarial pool, where detection is multi-layered and IP reputation is the first filter.
5. Parse and log Content-Signal lines now. Even before enforcement, a stored record of the signals you saw — and honored — is cheap insurance, especially with the EU AI Act's August 2 enforcement date creating legal weight behind machine-readable opt-outs.
6. Compare block rates before and after September 15. Expect managed challenges rather than clean 403s. Compare success rates per domain across the change so you can separate policy blocks from ordinary rotation and rate-limit issues.
7. Budget for licensed access. Where a target's data genuinely matters to your product, Pay Per Use may end up cheaper — and far more defensible — than an arms race.

What site owners should do

If your site is behind Cloudflare, check which of the three controls your zone is on now:

• Open your zone's Security settings and review the new AI traffic options; decide per category (search / agent / training) what you want to allow.
• If you used the old "Block AI bots" switch, you were moved to Disallow AI Training. To stay out of AI training but stay in search, keep it that way: choosing Block for training also removes you from Google, Apple and Bing search.
• If you monetize content, look at Pay Per Use as a third option between "open" and "blocked."
• If you rely on AI-search referrals for discovery, be careful about blanket-blocking Search — under the new taxonomy it remains the one default-allowed category for a reason.

Where proxies fit after September 15

An honest assessment, since this is a proxy provider's blog: Cloudflare's new system governs declared bots — crawlers that identify themselves and want to be let in legitimately. It does not change the physics of undeclared automation, but it does raise the stakes around it. As default blocking pushes more AI-adjacent collection out of the declared lane, more of it flows through headless browsers on residential and ISP IPs, which in turn sharpens the premium on IP quality: datacenter ranges are the first thing purpose-based filtering writes off, while residential proxies and static ISP proxies keep working precisely because they are indistinguishable from the human traffic that ad-supported pages exist to serve. (If you're weighing the trade-offs, start with our datacenter vs residential comparison.)

Two practical notes. First, the Forwarded: for=...;use=... transitive-trust proposal explicitly contemplates bots operating through proxy layers — declared agent traffic routed via proxies is part of the design, not a loophole. Second, whichever lane you choose, verify your infrastructure before you rely on it: our free proxy checker will tell you what your pool actually looks like from the outside.

Update, September 2026: Cloudflare wasn't the only gatekeeper to tighten access that month. From around September 13, Google began blocking rank trackers and SERP scrapers far more effectively, on top of the google.com/goto redirects it confirmed on August 26. We cover what changed, and what it did to our own Search Console data, in Google's September 2026 crackdown on SERP scrapers.

Sources

• Cloudflare: Content Independence Day — giving users choice over AI crawlers (July 1, 2026)
• TechCrunch: Cloudflare's new policy pushes AI companies to pay for publishers' content (July 1, 2026)
• The Register: Cloudflare to block 'cynical' search-and-scrape bots from ad-supported web pages (July 1, 2026)
• Cloudflare Docs: Web Bot Auth
• Cloudflare: Have it both ways: stay discoverable in search while disallowing AI training (September 15, 2026)
• Cloudflare: Say it once: introducing Bot Preference Sync (updated September 15, 2026)
• Cloudflare press release: Cloudflare Helps End the Search-or-AI-Training Tradeoff (September 15, 2026)

Published July 20, 2026. Updated September 27, 2026 with what took effect on September 15.