SOURCES · EPISODE 1

Is the deal that built the web breaking?

Research as of 2026-09-27. Every factual claim in the video, with the source we used. Numbers are as the source states them; where we did the arithmetic, or the source reports another source, we say so.

Cold open

• Over the 12 weeks to 26 September 2026, Cloudflare Radar showed Google crawling about 4.8 pages per referral, OpenAI about 315, Perplexity about 939 and Anthropic about 1,046; OpenAI's and Anthropic's ratios fell from the previous 12 weeks, Perplexity's rose about sixfold.
  Cloudflare Radar, AI Insights: Crawl-to-refer ratio (live data, 12-week window), 2026-09-27 · https://radar.cloudflare.com/ai-insights?dateRange=12w · accessed 2026-09-27

The deal

• Cloudflare's CEO describes the old bargain as simple: let Google copy your content for search, and Google sends you traffic.
  Cloudflare Blog (Matthew Prince), 'Content Independence Day: no AI crawl without compensation!', 2025-07-01 · https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/ · accessed 2026-09-27
• Microsoft also says the open web was built on an implicit exchange: publishers made content accessible and search helped people find it.
  Microsoft Advertising Blog, 'Building Toward a Sustainable Content Economy for the Agentic Web', 2026-02-03 · https://about.ads.microsoft.com/en/blog/post/february-2026/building-toward-a-sustainable-content-economy-for-the-agentic-web · accessed 2026-09-27
• The robots.txt convention, which lets a website ask crawlers to stay out, was agreed on 30 June 1994 by robot authors on a mailing list, and nobody enforces it.
  robotstxt.org, 'A Standard for Robot Exclusion' (Martijn Koster), 1994-06-30 · https://www.robotstxt.org/orig.html · accessed 2026-09-27

What changed

• Google launched the Knowledge Graph on 16 May 2012, so that its results page could show a summary of key facts about people, places and things.
  Google, The Keyword blog (Amit Singhal), 'Introducing the Knowledge Graph: things, not strings', 2012-05-16 · https://blog.google/products/search/introducing-knowledge-graph-things-not/ · accessed 2026-09-27
• Google began rolling out AI Overviews, AI-written answers at the top of search results, to everyone in the US on 14 May 2024.
  Google, The Keyword blog, 'Generative AI in Search: Let Google do the searching for you', 2024-05-14 · https://blog.google/products/search/generative-ai-google-search-may-2024/ · accessed 2026-09-27
• In a Pew study of 900 US adults' browsing in March 2025, people clicked a regular search result on 8% of Google visits that showed an AI summary, versus 15% when there was none, and clicked a link inside the summary on just 1%.
  Pew Research Center (Athena Chapekis, Anna Lieb), 'Google users are less likely to click on links when an AI summary appears in the results', 2025-07-22 · https://www.pewresearch.org/short-reads/2025/07/22/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results/ · accessed 2026-09-27
• On 20 May 2025 Google opened AI Mode, a chatbot-style search tab, to everyone in the US.
  Google, The Keyword blog, 'AI Mode in Google Search: Updates from Google I/O 2025', 2025-05-20 · https://blog.google/products/search/google-search-ai-mode-update/ · accessed 2026-09-27
• Cloudflare says Google still accounts for about 88% of referral traffic on its network, and that because one Google crawler serves both search and AI, sites cannot block Google's AI without leaving its search.
  Cloudflare Blog, 'Content Independence Day, one year on: building the business model for the agentic Internet', 2026-07-01 · https://blog.cloudflare.com/agentic-internet-bot-report/ · accessed 2026-09-27
• At Google I/O on 19 May 2026, Google said AI Overviews had over 2.5 billion monthly users and AI Mode over 1 billion.
  Google, The Keyword blog, Sundar Pichai's remarks at Google I/O 2026, 2026-05-19 · https://blog.google/innovation-and-ai/sundar-pichai-io-2026/ · accessed 2026-09-27
• The Reuters Institute notes that in ChatGPT, links to sources are provided in the form of a small 'citations' button.
  Reuters Institute (Nic Newman), 'Journalism, media, and technology trends and predictions 2026', 2026-01-12 · https://reutersinstitute.politics.ox.ac.uk/journalism-media-and-technology-trends-and-predictions-2026 · accessed 2026-09-27

The numbers

• In a Pew study of 900 US adults' browsing in March 2025, people clicked a regular search result on 8% of Google visits that showed an AI summary, versus 15% when there was none, and clicked a link inside the summary on just 1%.
  Pew Research Center (Athena Chapekis, Anna Lieb), 'Google users are less likely to click on links when an AI summary appears in the results', 2025-07-22 · https://www.pewresearch.org/short-reads/2025/07/22/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results/ · accessed 2026-09-27
• Ahrefs found that by December 2025, when an AI Overview appeared, the top-ranked result got about 58% fewer clicks than it would otherwise be expected to.
  Ahrefs Blog (Ryan Law), 'Update: AI Overviews Reduce Clicks by 58%', 2026-02-04 · https://ahrefs.com/blog/ai-overviews-reduce-clicks-update/ · accessed 2026-09-27
• Chartbeat data for the Reuters Institute show Google search traffic to over 2,500 news sites fell by a third worldwide between November 2024 and November 2025, and by 38% in the US.
  Reuters Institute (Nic Newman), 'Journalism, media, and technology trends and predictions 2026', 2026-01-12 · https://reutersinstitute.politics.ox.ac.uk/journalism-media-and-technology-trends-and-predictions-2026 · accessed 2026-09-27
• Chartbeat's March 2026 report said publishers' total pageviews in 2025 were down only about 6% from 2024, even though pageviews from Google Search fell 34%. unverified
  Nieman Lab (Laura Hazard Owen), 'AI sources like ChatGPT account for less than 1% of publishers' pageviews, Chartbeat says' (reported by this source), 2026-03-18 · https://www.niemanlab.org/2026/03/ai-sources-like-chatgpt-account-for-less-than-1-of-publishers-pageviews-chartbeat-says/ · accessed 2026-09-27

The other side

• Alphabet's revenue from Search and its other own properties grew 17% to $63.3 billion in April to June 2026, and Google says searches reached an all-time high.
  Alphabet Inc., 'Alphabet Announces Second Quarter 2026 Results' (Exhibit 99.1, SEC), 2026-07-22 · https://www.sec.gov/Archives/edgar/data/1652044/000165204426000066/googexhibit991q22026.htm · accessed 2026-09-27
• Google says total organic clicks from Search to websites were 'relatively stable' year over year in 2025, and that it now sends 'slightly more quality clicks' than a year earlier. contested
  Google, The Keyword blog (Liz Reid), 'AI in Search is driving more queries and higher quality clicks', 2025-08-06 · https://blog.google/products-and-platforms/products/search/ai-search-driving-more-queries-higher-quality-clicks/ · accessed 2026-09-27
• Google said the Pew study used a 'flawed methodology and skewed queryset', and that it sends billions of clicks to websites daily without significant drops in aggregate traffic.
  The Register, 'Google AI Overviews are killing the web, Pew study shows (again)' (reported by this source), 2025-07-22 · https://www.theregister.com/2025/07/22/google_ai_overviews_suppress_search/ · accessed 2026-09-27
• Clicks were falling even where no AI answer appeared: for Ahrefs' keywords without an AI Overview, the top result's click rate fell from 7.6% to 3.9% between December 2023 and December 2025.
  Ahrefs Blog (Ryan Law), 'Update: AI Overviews Reduce Clicks by 58%', 2026-02-04 · https://ahrefs.com/blog/ai-overviews-reduce-clicks-update/ · accessed 2026-09-27
• Anthropic says all three of its web bots honour robots.txt, and that it runs separate bots for model training (ClaudeBot), search (Claude-SearchBot) and fetching pages when a user asks (Claude-User), each of which sites can block separately.
  Claude Help Center (Anthropic), 'Does Anthropic crawl data from the web, and how can site owners block the crawler?', 2026-04-07 · https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler · accessed 2026-09-27
• OpenAI says it runs GPTBot for training and OAI-SearchBot for ChatGPT search as independent robots.txt settings, but that for ChatGPT-User, which fetches pages when a user asks, robots.txt rules may not apply.
  OpenAI Developers, 'Overview of OpenAI Crawlers', not stated (live page) · https://developers.openai.com/api/docs/bots · accessed 2026-09-27
• Perplexity says its crawler PerplexityBot is for search results and not for training AI models, and that its Perplexity-User fetcher, which visits pages when a user asks, generally ignores robots.txt.
  Perplexity docs, 'Perplexity Crawlers', not stated (live page) · https://docs.perplexity.ai/docs/resources/perplexity-crawlers · accessed 2026-09-27
• Cloudflare linked the fall in Anthropic's crawl-to-refer ratio in 2025 to Claude's web search, which added citations with clickable links that could send readers to websites.
  Cloudflare Blog (João Tomé), 'The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals', 2025-08-29 · https://blog.cloudflare.com/crawlers-click-ai-bots-training/ · accessed 2026-09-27
• Cloudflare suggested OpenAI's crawl-to-refer ratio came down in 2025 partly because more people used ChatGPT search, whose answers link back to the source websites.
  Cloudflare Blog, 'The 2025 Cloudflare Radar Year in Review: The rise of AI, post-quantum, and record-breaking DDoS attacks', 2025-12-15 · https://blog.cloudflare.com/radar-2025-year-in-review/ · accessed 2026-09-27
• Similarweb estimates AI platforms sent over 1.1 billion visits to websites in June 2025, up 357% from a year earlier.
  Similarweb press release, 'AI Discovery Surges: Similarweb's 2025 Generative AI Report Says', 2025-12-02 · https://ir.similarweb.com/news-events/press-releases/detail/138/ai-discovery-surges-similarwebs-2025-generative-ai-report-says · accessed 2026-09-27
• For news sites tracked by Chartbeat, Google still sent 500 times as many referrals as ChatGPT from search alone, and 1,300 times as many counting Google Discover.
  Reuters Institute (Nic Newman), 'Journalism, media, and technology trends and predictions 2026', 2026-01-12 · https://reutersinstitute.politics.ox.ac.uk/journalism-media-and-technology-trends-and-predictions-2026 · accessed 2026-09-27

The pushback

• On 1 July 2025 Cloudflare started blocking AI crawlers by default for new domains, asking each new site whether to allow them, and opened a 'pay per crawl' beta that lets sites charge crawlers.
  Cloudflare press release, 'Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large; Permission-Based Approach Makes Way for A New Business Model', 2025-07-01 · https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/ · accessed 2026-09-27
• Cloudflare announced that from 15 September 2026 new domains would by default block AI training crawlers and AI agents on pages that show ads, while still allowing search crawlers, and that crawlers doing both search and training, such as Googlebot, would be blocked wherever a site blocks training.
  Cloudflare Blog (Jin-Hee Lee, Bryan Becker), 'Your site, your rules: new AI traffic options for all customers', 2026-07-01 · https://blog.cloudflare.com/content-independence-day-ai-options/ · accessed 2026-09-27
• Cloudflare's 1 July 2026 press release says the 15 September defaults also apply to existing free-plan customers who have not changed their settings, and that mixed crawlers that do not let sites choose between search, agent use and training will be blocked on all pages with ads.
  Cloudflare press release, 'Cloudflare Allows the Agentic Internet to Flourish with a Simple Philosophy: Your Content, Your Rules', 2026-07-01 · https://www.cloudflare.com/press/press-releases/2026/cloudflare-allows-the-agentic-internet-to-flourish-with-a-simple-philosophy-your-content-your-rules/ · accessed 2026-09-27
• On 3 June 2026 the UK competition regulator ordered Google to let publishers keep their content out of AI Overviews, AI Mode and the training of its wider AI such as Gemini, and to credit sources with clear links.
  UK Competition and Markets Authority, press release 'CMA secures fairer deal for publishers and improves Google search services in UK', 2026-06-03 · https://www.gov.uk/government/news/cma-secures-fairer-deal-for-publishers-and-improves-google-search-services-in-uk · accessed 2026-09-27
• Google's Search Console now has a switch that removes a site from AI Overviews and AI Mode without affecting its normal ranking, and it reached all websites worldwide on 31 August 2026.
  Google Search Console Help, 'Search generative AI control', 2026-08-31 · https://support.google.com/webmasters/answer/16908024?hl=en · accessed 2026-09-27
• In June 2026 Google said it was piloting a new way to partner with websites whose content helps ground its AI answers, alongside a paid News AI pilot with more than 200 publications.
  Google Public Policy (Markham Erickson), 'Google’s ongoing commitment to supporting the information ecosystem', 2026-06-18 · https://publicpolicy.google/article/supporting-information-ecosystem/ · accessed 2026-09-27
• Digiday reported on 14 September 2026 that Google is testing an 'AI contribution pilot' that pays some websites through Search Console when their content contributes significantly to answers in the Gemini app, AI Overviews and AI Mode; Google confirmed it is an early-stage learning pilot. unverified
  Digiday (Jessica Davies, Sara Guaglione), 'Google rolls out pay-per-value AI licensing program to publishers' (reported by this source), 2026-09-14 · https://digiday.com/media/google-rolls-out-pay-value-ai-licensing-program-to-publishers/ · accessed 2026-09-27
• On 9 December 2025 the European Commission opened a formal antitrust investigation into whether Google uses publishers' content for AI Overviews and AI Mode without fair payment or a real way to refuse.
  European Commission press release, 'Commission opens investigation into possible anticompetitive conduct by Google in the use of online content for AI purposes', 2025-12-09 · https://ec.europa.eu/commission/presscorner/detail/en/ip_25_2964 · accessed 2026-09-27

What we don't know

• Ahrefs found that by December 2025, when an AI Overview appeared, the top-ranked result got about 58% fewer clicks than it would otherwise be expected to.
  Ahrefs Blog (Ryan Law), 'Update: AI Overviews Reduce Clicks by 58%', 2026-02-04 · https://ahrefs.com/blog/ai-overviews-reduce-clicks-update/ · accessed 2026-09-27
• Chartbeat data for the Reuters Institute show Google search traffic to over 2,500 news sites fell by a third worldwide between November 2024 and November 2025, and by 38% in the US.
  Reuters Institute (Nic Newman), 'Journalism, media, and technology trends and predictions 2026', 2026-01-12 · https://reutersinstitute.politics.ox.ac.uk/journalism-media-and-technology-trends-and-predictions-2026 · accessed 2026-09-27
• Clicks were falling even where no AI answer appeared: for Ahrefs' keywords without an AI Overview, the top result's click rate fell from 7.6% to 3.9% between December 2023 and December 2025.
  Ahrefs Blog (Ryan Law), 'Update: AI Overviews Reduce Clicks by 58%', 2026-02-04 · https://ahrefs.com/blog/ai-overviews-reduce-clicks-update/ · accessed 2026-09-27
• Cloudflare warns its crawl-to-refer ratios may overstate the imbalance, because visits sent from AI companies' own apps carry no referrer tag and go uncounted.
  Cloudflare Blog, 'The crawl before the fall… of referrals: understanding AI's impact on content providers', 2025-07-01 · https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar/ · accessed 2026-09-27
• Google says total organic clicks from Search to websites were 'relatively stable' year over year in 2025, and that it now sends 'slightly more quality clicks' than a year earlier. contested
  Google, The Keyword blog (Liz Reid), 'AI in Search is driving more queries and higher quality clicks', 2025-08-06 · https://blog.google/products-and-platforms/products/search/ai-search-driving-more-queries-higher-quality-clicks/ · accessed 2026-09-27
• In July 2025, on Cloudflare's network, Google crawled about 5 pages for every visit it referred back to a website, OpenAI about 1,091 and Anthropic about 38,066.
  Cloudflare Blog (Joao Tome), 'The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals', 2025-08-29 · https://blog.cloudflare.com/crawlers-click-ai-bots-training/ · accessed 2026-09-27
• Over the 12 weeks to 26 September 2026, Cloudflare Radar showed Google crawling about 4.8 pages per referral, OpenAI about 315, Perplexity about 939 and Anthropic about 1,046; OpenAI's and Anthropic's ratios fell from the previous 12 weeks, Perplexity's rose about sixfold.
  Cloudflare Radar, AI Insights: Crawl-to-refer ratio (live data, 12-week window), 2026-09-27 · https://radar.cloudflare.com/ai-insights?dateRange=12w · accessed 2026-09-27

What to watch

• The UK rules take effect on 3 December 2026, page-by-page opt-outs by 3 March 2027, and Google's first compliance report to the CMA is due on 3 January 2027.
  UK Competition and Markets Authority, 'Publisher CR notice' (PDF), 2026-06-03 · https://assets.publishing.service.gov.uk/media/6a1ec73a59fb7a60f827f64e/Publisher_CR_Notice.pdf · accessed 2026-09-27

Corrections

None yet. Corrections are listed here and in the video description.

Corrections

None so far. If you find an error, write to hello@radar.yt.