Vendor / platform

Per-vendor (mixed sources)

Catch-all bucket for AI crawler permission docs that vary per vendor (OpenAI, Anthropic, Google, Common Crawl). Each vendor publishes its own user-agent and opt-out conventions.

Short name: Per-vendor
Scope: Vendor / platform
Homepage: developers.openai.com
Standards governed: 1

Standards governed by Per-vendor

AI crawler permissions robots.txt convention global AI-readiness Explicit allow/disallow rules for GPTBot, ClaudeBot, PerplexityBot, and friends. Default-deny means missing AI citations; default-allow means free training data.