WebExtractor.app
Crawl & URL Extractor

Robots.txt & XML Sitemap Analyzer

Parse robots.txt directives, test any path against User-Agent rules, automatically discover XML sitemaps, and export extracted URLs to CSV.

Why Use WebExtractor for Robots & Sitemap?

Live Path Allow/Disallow Tester

Test any URL path (like /blog/post or /admin) against parsed robots.txt Allow/Disallow rules instantly.

Auto Sitemap Discovery

Reads Sitemap: directives from robots.txt and falls back to /sitemap.xml automatically.

Sitemap Index Support

Handles both standard URL sitemaps and nested sitemap index files with lastmod dates.

One-Click CSV / TXT Export

Export extracted sitemap URLs for Screaming Frog, GSC audits, or migration mapping.

How It Works

1

Enter Domain or Sitemap URL

Provide a root domain (example.com) and an optional path to test.

2

Inspect Rules & Sitemap URLs

Review parsed User-Agent groups, test path status, and preview discovered URLs.

3

Export URLs to CSV

Download the clean URL list for your SEO audit.

Frequently Asked Questions

How does robots.txt Allow vs. Disallow precedence work?+

Googlebot uses longest-match precedence: when both an Allow and Disallow rule match a URL path, the more specific (longer) pattern wins. If lengths tie, Allow wins.

Does blocking a URL in robots.txt remove it from Google?+

Not always. Google can still index a blocked URL without crawling its content if other sites link to it. Use a noindex meta tag (on a crawlable page) to reliably deindex a page.

More WebExtractor Tools