Google, Microsoft, OpenAI and Anthropic to honour a single robots.txt opt-out for AI training

A common opt-out turns training-data consent from a patchwork of crawler-specific rules into one control that publishers and regulators can point to. Publishers and rights holders should audit their robots.txt files and decide their position before the 2027 start dates. The signal does not create payment, so licensing disputes will move to other forums, and lawmakers weighing consent mandates now have an industry standard to measure against.

What Happened

An IETF draft by Kevin Kelley of Anthropic, published on Friday October 30, proposes a shared signal, and Google, OpenAI and Microsoft announced the same day that they would support it. The signal lets a website keep its content out of generative AI training with a single rule. Google and Microsoft start in 2027, and Microsoft says the signal creates no licensing or payment deals.

Why It Matters

A common opt-out turns training-data consent from a patchwork of crawler-specific rules into one control that publishers and regulators can point to. Publishers and rights holders should audit their robots.txt files and decide their position before the 2027 start dates. The signal does not create payment, so licensing disputes will move to other forums, and lawmakers weighing consent mandates now have an industry standard to measure against.

Sources

• The Next Web (opens in a new tab): Google, Microsoft, OpenAI and Anthropic back one AI training opt-out (2026-10-31)

More From This Briefing

• Tampered Claude Code extension served on Cursor marketplace with persistent SSH backdoor; Anthropic denies breach
• Aramco takes 49% of Saudi AI company HUMAIN with $2.6bn cash and AI assets
• CISA adds two exploited Ivanti Connect Secure vulnerabilities to its KEV catalog
• Willis Re completes acquisition of US reinsurance broker BMS Re US

More on Financial Markets

• Tech stocks sell off as investors demand proof that AI spending is producing returns (1 November 2026)
• OpenAI cuts funding round target to up to $20bn at a reported $1.25tn valuation (1 November 2026)
• Alliant Energy targets 1 GW of further data-centre contracts within six months, raises capex plan 19% (1 November 2026)
• Microsoft Azure grows 48% and commercial backlog rises 73% to $677bn in FY27 Q1 (31 October 2026)

All Financial Markets coverage →

Key Terms

• AI audit: An AI audit is an independent examination of an AI system's controls, records and outcomes against defined criteria such as a law, a...

Filed under Financial Markets, AI Governance and Regulation, AI in Financial Services, Anthropic, Google, Microsoft, OpenAI.

This story is part of the Laurenceo Intelligence briefing for 2 November 2026, a daily sourced briefing on AI, technology and enterprise risk published in Bermuda by Laurenceo, founded by Jorai Laurenceo. It is produced by an automated research pipeline and checked before publication; how this is made. General information only, not legal, regulatory or financial advice.