About crawler.sh
crawler.sh is a versatile tool that crawls websites quickly from a local machine, converting the data into structured formats for SEO audits, content archiving, and sitemaps.
Feature Highlights
High-speed site crawling with configurable settings
Content extraction to Markdown for consistent use
Automated SEO analysis with 16 checks per URL
Multiple export formats for various data needs
Desktop dashboard and CLI with real-time features
Local-first, privacy-friendly design
Use Cases
Auditing client sites for SEO agencies and consultants
Monitoring corporate sites for in-house SEO teams
Automating crawls for developers and technical SEOs
Converting articles to Markdown for content operations
Used by product teams for crawling staging environments
Adopted by researchers for structured text corpora in ML experiments