If a page is not indexed, it does not exist in search results. That statement sounds obvious, but the root causes of indexation failures are varied enough that diagnosing them correctly requires methodical investigation rather than guesswork. I have audited sites where entire subdirectories were being blocked by a single robots.txt rule the client never intended to write.

How to Check What Is Indexed

The fastest way to get a directional count is the site: operator in Google Search: site:yourdomain.com. The number is not precise — Google has said so explicitly — but it gives a ballpark. A more reliable view comes from Google Search Console's Indexing report, which shows indexed pages, excluded pages, and the specific reason Google is not indexing each excluded URL. I always start there before running any tool-based crawl.

  • Check the Index Coverage report and sort by the most common exclusion reason
  • Use URL Inspection on specific pages to see the last crawl date, canonical, and whether the page is eligible to appear
  • Compare the crawl tool's total page count against the Search Console indexed count — large gaps indicate a problem
  • Request indexing via Search Console for important pages after fixing the underlying issue, but do not spam the request button

Robots.txt and Noindex Tags

The two most common self-inflicted indexation problems are misconfigured robots.txt disallow rules and noindex meta tags left in place after development or migration. I have seen staging environments pushed to production with a blanket Disallow: / that was never removed. Google will not index a page its crawler cannot visit, and it will not index a page with a valid noindex directive. These are quick wins when caught early and expensive mistakes when caught months later.

  • Test robots.txt using the Google Search Console Robots.txt Tester before making changes
  • Crawl the site for noindex meta tags and X-Robots-Tag headers separately — CMSes sometimes add them at the template level
  • Check that noindex is not being set programmatically based on query parameters or user-agent logic that was never removed

Here is what a common accidental staging robots.txt looks like, and what the corrected production version should say:

# Staging (WRONG — blocks everything)
User-agent: *
Disallow: /

# Production (correct — allows all crawling, blocks only admin)
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xml

If a site launched with the first version and it was not caught for several weeks, Google will have recorded those pages as crawled but blocked. Even after fixing the robots.txt, re-crawling and re-indexing takes time. The URL Inspection tool in Search Console lets you force a recrawl of specific URLs, which can speed up recovery for high-priority pages.

Canonicalization Problems

When a page has a canonical tag pointing to a different URL, Google treats the canonical destination as the preferred version and typically will not index the pointing page separately. Canonical tags can cause indexation to collapse in unexpected ways: if the canonical on a given page points to a URL that itself has another canonical, you have a chain that Google may or may not follow correctly. Self-referencing canonicals are the safe default, and any cross-pointing canonical should be intentional.

  • Audit canonical tags across page templates, not just spot-checking individual URLs
  • Look for canonicals set in the HTTP header in addition to the HTML head — both are valid and both count
  • Watch for pagination pages incorrectly canonicalized to page one
  • Verify that hreflang alternate tags reference the canonical version of each URL, not a redirect or alternate path

Crawl Budget and Thin Content

On large sites, Google will not crawl every URL on every visit. If your site produces thousands of low-value URLs through faceted navigation, session parameters, or auto-generated tag pages, crawl budget gets spread thin and important pages may be crawled infrequently. Google also exercises discretion about what it indexes: thin pages, near-duplicate pages, and pages with very low word counts may be crawled but deliberately excluded from the index. Cleaning up URL parameters and consolidating thin content improves both crawl efficiency and indexation rates.

Crawl-Indexed Gap Analysis

One of the most useful diagnostics in a technical audit is comparing your total crawlable URL count against your indexed page count in Search Console. I call this the crawl-indexed gap. On a healthy site, the gap should be small and explainable — login pages, thank-you pages, admin paths, and intentionally noindexed URLs account for the difference. A large unexplained gap means Google is encountering pages it can crawl but is choosing not to index, which is usually a quality signal.

The Search Console Index Coverage report breaks the excluded pages into reasons:

  • Crawled — currently not indexed: Google crawled the page but decided not to include it. This is almost always a content quality issue — thin, thin-duplicate, or low-E-E-A-T content.
  • Discovered — currently not indexed: Google knows the URL exists but has not yet crawled it. This is a crawl budget or crawl scheduling issue — improve internal linking to the affected pages.
  • Excluded by 'noindex' tag: Intentional or accidental noindex. Check whether these are supposed to be excluded.
  • Alternate page with proper canonical tag: Google found a canonical pointing elsewhere and is deferring. Verify the canonical is correct.

Rendering and JavaScript Indexation

A less obvious indexation problem affects sites where important content is loaded via JavaScript. Googlebot renders JavaScript, but it does so in a second wave of processing that can lag crawl by days or weeks. If your page content — especially the main body text, headings, or internal links — is injected by client-side JavaScript, there is a window where Google crawls the page but indexes only the pre-render shell. Using server-side rendering or pre-rendering for content that needs to be indexed is the reliable fix. You can test what Google actually sees with the Live URL Test in the URL Inspection tool, which shows the rendered DOM.

Hreflang and Indexation for Multilingual Sites

Sites that serve content in multiple languages add another layer of indexation complexity through hreflang tags. When hreflang is implemented incorrectly, Google may index the wrong language version for a given region, or it may treat the alternate-language pages as near-duplicates and consolidate them unexpectedly. The most common mistake I see is hreflang tags that reference redirect URLs or non-canonical versions of the page instead of the final canonical destination. Every hreflang href value should point to the same URL that appears in the canonical tag for that language version.

If you are running a multilingual site and seeing indexation problems in Search Console for specific language variants, cross-check the hreflang implementation against the canonical tags on the same pages. Inconsistencies between the two are a frequent root cause of indexation failures on international sites.

Indexation problems and duplicate content are closely related — canonical and noindex choices often stem from the same root cause. A complete technical picture requires examining both together, which is exactly what a structured website audit covers.