SEO

Robots meta + X-Robots-Tag

Page-level indexing controls — `noindex`, `nofollow`, `nosnippet`, `max-image-preview`. Distinct from robots.txt, which only blocks crawling.

Authority: Google
Version: Google Search Central
Jurisdiction: Global
Source: developers.google.com
Last reviewed: 2026-04-28
Last verified: pending

What it is

`<meta name="robots">` directives in the HTML head, plus the `X-Robots-Tag` HTTP response header (used for non-HTML files like PDFs or image responses). Both share the same vocabulary — `noindex`, `nofollow`, `noarchive`, `nosnippet`, `max-snippet`, `max-image-preview`, `max-video-preview`, `noimageindex`, etc.

Why it matters

Crawl-blocked URLs (robots.txt) can still appear in search results without snippets. Only `noindex` actually keeps a page out of the index — and many sites get this backwards, blocking crawl on URLs they want fully removed. X-Robots-Tag is the only way to deindex a PDF or image without removing the file.

Who it applies to

Every site with internal pages, staging environments, or non-HTML assets that shouldn't be in search.

How WQI scores it

Web Quality Index considers this standard satisfied when the supporting factor passes.

#	Factor	Status
14	Sitemap.xml + robots.txt presence	live

Related standards

See also: robots/sitemap , Canonical

Standards that share factors with this one

Auto-computed from overlapping factor tickets in satisfiedBy, excluding standards already listed under "See also" above. Strong overlap suggests these standards rise and fall together when sites are scored.

Google Search Essentials 1 shared factor SEO Google's umbrella ranking-and-eligibility guide — replaced the old `Webmaster Guidelines` in 2022. Three pillars: technical requirements, spam policies, and the helpful-content / E-E-A-T quality bar.

Other references

guidance MDN — Meta robots

Examples

X-Robots-Tag for a PDF http

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex, nofollow

PDFs can't carry meta tags, so X-Robots-Tag is the only deindex path short of removing the file.