robots.txt is a text file placed at the root of a site to communicate crawl preferences. A Sitemap line inside robots.txt can also tell crawlers where to find sitemap files.
The Sitemap directive helps discovery
A sitemap placed at /sitemap.xml may be discovered without any extra hint. But many CMSs, multilingual sites, and large sites use multiple sitemap URLs or non-default paths. Listing them in robots.txt gives crawlers a clearer starting point.
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlIt is separate from Disallow
Disallow rules describe paths you do not want crawled. Sitemap directives point to URL lists you want crawlers to discover. They both affect crawling, but they serve different purposes.
What to check
- robots.txt is available at the site root
- Sitemap values are absolute URLs
- Referenced sitemap files can be fetched
- HTTP/HTTPS and www variants are not mixed by mistake
- Multilingual or split sitemap files are not missing
- Old staging or preview URLs are not left behind
robots.txt and sitemap settings are easy to break during launches, CMS changes, and domain moves. The TOOLPOOL Sitemap Checker checks robots.txt Sitemap references and the actual sitemap documents together.