robots.txt + sitemap.xml

What it is

robots.txt (RFC 9309) controls crawler access at the host level; sitemap.xml (sitemaps.org) lists indexable URLs with metadata. Both live at well-known paths.

Why it matters

Without a sitemap, search engines have to discover every URL through links — slow and incomplete on large sites. Without robots.txt, you can't direct AI crawlers or block low-value paths.

Who it applies to

Every public-facing site.

How WQI scores it

Web Quality Index considers this standard satisfied when the supporting factor passes.

Related standards

See also

AI crawlers , Schema.org

Standards that share factors with this one

Auto-computed from overlapping factor tickets in satisfiedBy, excluding standards already listed under "See also" above. Strong overlap suggests these standards rise and fall together when sites are scored.

• Google Search Essentials 1 shared factor SEO Google's umbrella ranking-and-eligibility guide — replaced the old `Webmaster Guidelines` in 2022. Three pillars: technical requirements, spam policies, and the helpful-content / E-E-A-T quality bar.
• Robots meta + X-Robots-Tag 1 shared factor SEO Page-level indexing controls — `noindex`, `nofollow`, `nosnippet`, `max-image-preview`. Distinct from robots.txt, which only blocks crawling.

Other references

• spec sitemaps.org protocol
• guidance Google Search Central — robots.txt