methodology / SEO & Discoverability / #14

Sitemap.xml + robots.txt presence

#14 · Recommended · Web Quality · weighted · SEO & Discoverability · weight 1.3% · impl implemented · method v1.2.0

Web Quality factor

This factor is part of Web Quality — the weighted 0..100 score that sits above Web Standards. Its weight depends on what kind of site is being measured. Web Standards items take priority; this factor only enters the score once Web Standards passes.

Base weight: 1.0 applied to every site type unless overridden below
Why this weight: robots.txt + sitemap are how the site instructs crawlers. Polite and trivial to provide.

Per-site-type overrides

Site type	Weight	Δ vs base
Personal site	0.5	-0.5

Site types not listed inherit the base weight.

Same factor, two depths.

What we measure

These two files tell search engines what to crawl and what to skip. Missing them slows down indexing and may exclude important pages.

How to improve your score

Sitemap: WordPress generates one automatically (`/sitemap.xml` or `/sitemap_index.xml`). Robots.txt: place at site root with at minimum: `User-agent: *\nAllow: /\nSitemap: https://yourdomain.com/sitemap.xml`.

Facts

Ticket

WEBQ-14

When this applies

This platform manages robots.txt centrally and doesn't let site owners customize it.

Marked n/a when the detected platform doesn't support canControlRobotsTxt (e.g., Squarespace and Wix can't set custom HTTP headers, so factor #4 becomes n/a there).

Scoring

Scoring formulas are versioned with the methodology. The current method (v1.2.0) maps raw measurements to pass, warn, fail. Factor weights determine how much each contributes to the composite — see the methodology index for the full table.

Cited by these standards

Standards in the Standards Library whose satisfiedBy requirement tree references this factor. Each link goes to the standard's full entry — methodology, scope, and the other factors it relies on.

Google Search Essentials SEO global Google's umbrella ranking-and-eligibility guide — replaced the old `Webmaster Guidelines` in 2022. Three pillars: technical requirements, spam policies, and the helpful-content / E-E-A-T quality bar.
Robots meta + X-Robots-Tag SEO global Page-level indexing controls — `noindex`, `nofollow`, `nosnippet`, `max-image-preview`. Distinct from robots.txt, which only blocks crawling.
robots.txt + sitemap.xml SEO global Tell crawlers what to index and what to skip. The most basic site-discoverability hygiene.

Version history

Version	Change	Date
v1.2.0	Factor introduced. Status: live. Scoring impl: implemented.	2026-04-25

← back to methodology