This package get, fetch, crawl, sitemap pages recursively and fetch all links in between <loc> tag.
Collect links through the sitemap.xml or robots.txt
GoSitemap2Md is a Golang program that generates a sitemap URL in Markdown format and stores the URLs in a urls.json file for easy adding of new URLs. This tool simplifies the process of generating and maintaining a sitemap for your website.
a python script that crawls website sitemap in a very quick way with multi threading and extract, write SEO based data to CSV file
The Firecrawl Toolkit is the easiest way for developers to interact with web content through crawling, scraping, and mapping capabilities.
Add a description, image, and links to the sitemap-crawler topic page so that developers can more easily learn about it.
To associate your repository with the sitemap-crawler topic, visit your repo's landing page and select "manage topics."