XML sitemaps explained: create, submit and troubleshoot

A sitemap is a discovery and crawl-scheduling hint. Google's own documentation is explicit about this: submitting a sitemap doesn't guarantee that all the URLs in it will be crawled or indexed, and inclusion in a sitemap is not a ranking signal at all. It's a list of candidates, not a queue that Google works through in order.

The misconception worth killing early: 'my page isn't ranking, let me check if it's in the sitemap.' Being in the sitemap has nothing to do with ranking, and often nothing to do with indexing either if the page has other problems, thin content, a noindex tag, a canonical pointing elsewhere, low perceived quality. A sitemap gets a page in front of the crawler faster. It does not make Google like the page.

What a sitemap does concretely: it exposes URLs Google might not find through normal link crawling, it communicates lastmod dates so Google can prioritize re-crawling changed pages over static ones, and on very large sites it can be split by content type so you can diagnose indexing problems by section in Search Console rather than staring at one undifferentiated list.

A minimal sitemap is just this, one <url> block per page:

If the site runs on Yoast SEO, Rank Math, or All in One SEO, a sitemap is already being generated automatically at /sitemap_index.xml (Yoast, Rank Math) the moment the plugin is active, no separate setup step needed for the file itself. What actually needs attention is the settings that control what goes into it.

Check three things in the plugin's sitemap settings: that post types you don't want indexed (attachments, some custom post types, internal utility pages) are excluded from the sitemap, that noindexed pages are automatically dropped from it (all three major plugins do this by default, but verify after any bulk content changes), and that pagination and archive pages aren't bloating the file with low-value URLs.

A common WordPress mistake is running two SEO plugins at once, each generating its own sitemap at a different path, and submitting both to Search Console. Pick one plugin, deactivate the other's sitemap module if it lingers, and submit a single sitemap index.

Without a CMS plugin, you're either generating the file at build time (static sites, Next.js, Astro) or serving it dynamically from a route that queries your database of published URLs.

For a static or server-rendered site, the practical approach is a small script that runs on deploy: pull every canonical, indexable URL from your content source, write out the <url> blocks, and write the file to the public root. Most frameworks have a documented sitemap convention (a sitemap.xml route or a config-driven generator) rather than requiring you to hand-roll the XML each time.

Whatever the mechanism, the rule is the same: the sitemap should only contain URLs that return a 200 status, are canonical (not duplicates pointing elsewhere via rel=canonical), and are not noindexed. Anything else pollutes the file and wastes crawl attention on URLs that were never going to get indexed anyway.

A single sitemap file is capped at 50,000 URLs and 50MB uncompressed. Past that, split into multiple sitemap files and list them in a sitemap index file, which is itself just a list of sitemap URLs.

In Search Console, go to Sitemaps in the left sidebar, enter the path relative to your verified property (just 'sitemap.xml' or 'sitemap_index.xml', not the full URL), and submit. Google fetches it, and the status updates from 'Pending' to 'Success' or an error state, usually within a few minutes to a few hours.

Also reference the sitemap's absolute URL in robots.txt with a Sitemap: line. This isn't redundant with the Search Console submission, some crawlers other than Googlebot only discover sitemaps through robots.txt, and it's a one-line addition.

This means Google saw the URL in your sitemap but chose not to index it. This is the most common sitemap-related report and almost never means the sitemap itself is broken.

Start with URL Inspection on the specific URL. It will usually tell you why: low-value content relative to similar pages already indexed, a duplicate of a canonical Google chose elsewhere, or a page Google crawled but assessed as not worth storing. Check whether the page has thin or near-duplicate content compared to other pages on the site, whether internal links actually point to it (a page only reachable via the sitemap and nowhere else in your site's link structure is a weak signal), and whether the page returns useful content on first render, not just after client-side JavaScript finishes.

Resubmitting the sitemap does nothing here. The fix is improving the page or the links pointing at it, not resubmitting the file.

This one is unambiguous: the URL is in your sitemap, but the page itself carries a noindex directive, either in a meta robots tag or an X-Robots-Tag HTTP header. Google is telling you the two signals contradict each other and it's obeying the noindex.

Check the page's rendered HTML source for <meta name="robots" content="noindex">. If it's not there, check response headers for an X-Robots-Tag, this is often set at the server or CDN level and invisible in the page source, so view-source won't catch it, you need to inspect the actual HTTP response.

The most frequent cause is a staging environment's noindex setting migrating to production, common with WordPress's 'Discourage search engines' toggle under Settings > Reading getting left on after launch, or a CDN/reverse-proxy rule written for a staging subdomain that got applied site-wide. Fix the directive, then remove the URL from the sitemap only if it should actually stay out of the index, otherwise leave it in and let Google re-crawl once the noindex is gone.

Search Console reports errors like 'Couldn't fetch', 'General HTTP error', or 'Sitemap is not in a supported format' when it can't parse or reach the file at all. Work through these in order.

Fetch the sitemap URL directly in a browser and confirm it returns a 200 status and valid XML, not a 404, a redirect, or an HTML error page rendered with a 200 status (a common false-positive where the server returns a friendly error page instead of an actual 404). Validate the XML is well-formed: an unescaped ampersand in a URL, a missing closing tag, or wrong character encoding will break parsing entirely. Confirm the sitemap isn't blocked by robots.txt, a Disallow rule covering the sitemap's own path will stop Google from fetching it. Check that the file is served with an XML or text content type rather than something the fetcher rejects, and that it's under the 50MB/50,000-URL cap, oversized files can time out or fail silently depending on server configuration.

If the sitemap is dynamically generated, also check that it isn't accidentally gated behind authentication, a maintenance-mode redirect, or a bot-blocking rule meant for scrapers that's catching Googlebot too.

A sitemap degrades quietly. Pages get deleted without being removed from the file, canonical tags change without the sitemap being regenerated, and lastmod dates get bumped on every deploy regardless of whether content actually changed. That last one specifically trains Google to stop trusting your lastmod signal, so only update it when the page's substantive content changes.

Regenerate the sitemap on a schedule tied to content changes, not on a fixed timer disconnected from reality, and periodically diff the sitemap's URL count against your CMS's actual published-page count. A gap between the two numbers is usually the first sign something's misconfigured.