Keep deleted URLs out of an XML sitemap

An XML sitemap should follow the content that remains public and indexable. When an article is deleted or returned to draft, a manually maintained sitemap can keep advertising an obsolete URL. Generate entries from the same publication rules used by public routes, then test those rules with a small fixture.

Filter before serializing XML

This synthetic example contains one visible article and two records that should stay out of the sitemap. A deleted row is absent from the input entirely.

from xml.etree.ElementTree import Element, SubElement, tostring

rows = [
    {"slug": "public", "published": True, "noindex": False},
    {"slug": "draft", "published": False, "noindex": False},
    {"slug": "excluded", "published": True, "noindex": True},
]
root = Element("urlset", xmlns="http://www.sitemaps.org/schemas/sitemap/0.9")
for row in rows:
    if row["published"] and not row["noindex"]:
        node = SubElement(root, "url")
        SubElement(node, "loc").text = "https://example.com/blog/" + row["slug"]
xml = tostring(root, encoding="unicode")
assert "https://example.com/blog/public" in xml
assert "/draft" not in xml and "/excluded" not in xml

The XML library handles text escaping. In an application, also validate slugs and construct canonical URLs from a controlled origin. Avoid deriving sitemap origins from arbitrary request headers.

Only published, indexable records become sitemap entries; deleted records are absent from the input.

Test the deletion boundary

A useful integration test publishes a fixture article, confirms its URL is present, deletes that record, then fetches the sitemap again. Check the article route's HTTP response as well. Removing a sitemap entry while the route still returns public content does not remove that content from the web.

If a response cache sits in front of the generated file, inspect its freshness rules. The database can be correct while the served sitemap remains stale. Record the public response after a change rather than checking only the internal function that builds entries.

Treat discovery as a separate observation

Google describes sitemap URLs as discovery information and does not guarantee that submitted pages will be crawled or indexed. A deleted URL may remain visible in search for a while after the site changes. Keep the HTTP behavior, canonical metadata and internal links consistent so that later fetches convey the intended state.

Use a truthful modification timestamp tied to meaningful content changes. Rewriting every timestamp on every request creates misleading change signals. Reference: Google sitemap overview. Related publishing diagnostics appear in the engineering blog.