Skip to content
Company Atlas
Crawler

CompanyAtlasBot

Company Atlas observes publicly accessible corporate web pages with a clearly identified, polite crawler. This page is linked from its User-Agent string.

Identity

User-Agent: CompanyAtlasBot/0.1 (+https://www.company-atlas.co/bot; contact@spboucher.ai)

Requests originate from the MacLustr infrastructure operated by Simon-Pierre Boucher (Québec, Canada). Contact for any question, rate concern or removal request: contact@spboucher.ai.

What we collect

  • Publicly accessible pages of company websites: homepage, about, careers and job boards, newsroom, blog and feeds, products, pricing, leadership, locations, documentation, changelogs, legal/terms/privacy, security, investor relations, partners, customers, sitemaps.
  • Public structured data on those pages (JSON-LD, microdata, RSS/Atom) and public JSON endpoints that the page itself loads.
  • Text and structure only. Images, videos, fonts, binaries and tracking scripts are not downloaded by default.

What we never do

  • Never log in, never bypass authentication, paywalls, CAPTCHAs or bot challenges; a challenge page is recorded as a failure and the sensor backs off.
  • Never collect non-public or personal consumer data. People appear only in their public professional context on the company’s own pages.
  • Never access private networks, localhost or infrastructure endpoints.

How it behaves

  • Honours robots.txt (including Crawl-delay) for CompanyAtlasBot and *.
  • Per-domain concurrency of 1–2 and per-domain rate limits; adaptive schedules from every 5–15 minutes for very active pages to every 3–7 days for stable ones; conditional requests (ETag / Last-Modified) whenever the server supports them.
  • Retry budgets with exponential backoff; sustained 429/403 responses pause the domain.
  • Redirect and size caps; no endless pagination, calendars or faceted crawling.

Opting out

Add to your robots.txt:

User-agent: CompanyAtlasBot
Disallow: /

Or disallow only specific paths. Changes are picked up within 24 hours. To also remove already collected observations of your pages, write to contact@spboucher.ai from a company address; we suppress the surfaces and mark the history as unavailable rather than pretending it never existed.

Attribution

Every event on Company Atlas links to the public page it was detected on. See also the about page and the API terms.