Tech News
OpenAI, Google, Microsoft and Anthropic back shared anti-scraping signal
Oct 30, 2026 · Summarized by Fundy Tech from The Verge
The 30-second version
The companies plan to support a shared robots.txt signal for publishers to block AI scraping. It lets sites set AI training preferences once instead of separately.
Key points
- The signal is a proposed shared “policy token” in robots.txt.
- Anthropic’s Kevin Kelley proposed it through the IETF.
- Anthropic has already adopted it.
- Google and Microsoft plan support next year; OpenAI in coming months.
Why it matters
This gives websites a simpler way to control how AI companies use their content. For small publishers and businesses, it may reduce the need to manage separate rules for each crawler. It also shows the industry is moving toward shared standards for AI training preferences.
Who’s affected: Website owners and publishers
What to do
- Review your robots.txt settings.
- Decide whether you want to allow AI training.
- Watch for support from your website tools.
- Follow publisher and platform policy updates.
Original reporting
Not sure if this affects you?
Fundy Tech helps businesses and households across Atlantic Canada stay patched, protected and up to speed. Tell us what you’re running and we’ll tell you straight whether you need to act.