The sitemap format didn't start as a joint web standard — Google introduced it alone in June 2005 as an experimental way to surface URLs its crawler might otherwise miss; only in November 2006 did Yahoo and Microsoft agree to support the same XML schema, turning a Google-only file into the shared sitemaps.org protocol used today.
Which fields actually matter
The <loc> tag with the page's absolute address is the only required field. <lastmod> is worth setting accurately, since Google does use it to help prioritize recrawling — unlike <priority> and <changefreq>, which Google has publicly said it largely ignores.
A shared spec, not a Google-only file
Because three separate search engines agreed to read the same format, sitemaps.org could define autodiscovery through a Sitemap: line in robots.txt — a site only needs to publish one file, and any crawler that follows the joint spec can find it without a manual submission per engine.
Absolute URLs and escaping special characters
Every address in the file must be absolute (protocol and domain included), and characters like the ampersand need to be written as an XML entity (&) — otherwise the file isn't valid XML and a crawler may discard the entire sitemap rather than skip the one broken entry.
What this is for
- Turn a plain list of site URLs into a correctly formatted sitemap.xml without hand-writing XML.
- Avoid the formatting mistakes that cause a search engine to reject the whole file.
- Regenerate the file after adding new pages instead of editing markup by hand.
Why a newer alternative exists alongside sitemaps
Sitemaps work by invitation: a crawler still has to come and fetch the file on its own schedule. IndexNow, launched by Bing and Yandex in 2021, flips that model — a site pings an endpoint the moment a page changes, and participating engines can fetch it almost immediately. Google hasn't adopted IndexNow, so for Google specifically, a well-formed sitemap plus normal crawling is still the mechanism that matters.