CavalryAuditBot

CavalryAuditBot is a website crawler operated by Cavalry Growth Systems. It reviews public pages of small-business websites, mainly home-service contractors, for our own website audits.

Only public web pages: the homepage, pages found through the site's own links and sitemap, the robots.txt file, and sitemaps. At most 25 pages per audit, and only on the website's own domain.

• It follows robots.txt and skips any page robots.txt disallows.
• It makes at most 2 requests at a time, at least 1 second apart for the same website, or slower if robots.txt sets a Crawl-delay (up to 10 seconds).
• It retries at most once, and only after a network error or a "server busy" response.
• It stops each audit after 10 minutes.
• It never submits forms, never logs in, never uses cookies, never accesses private pages, and never runs security or vulnerability scans.

It identifies itself with the user agent:

CavalryAuditBot/1.0 (+https://bot.cavalrygrowthsystems.com)

Add these two lines to your website's robots.txt file:

User-agent: CavalryAuditBot
Disallow: /

CavalryAuditBot checks robots.txt before every audit.

Email crawler@cavalrygrowthsystems.com. Please include the address of your website in your email.