Crawler
EnforceGateBot
EnforceGateBot is the web crawler behind EnforceGate's managed category feeds. It visits publicly reachable websites and categorizes them (news, shopping, gambling, adult, malware, phishing, and similar categories) so that EnforceGate deployments can allow or block by category. It is a safety/classification crawler: it does not index for search, serve ads, or resell content.
It identifies itself with this User-Agent on every request:
Mozilla/5.0 (compatible; EnforceGateBot/0.1; +https://www.enforcegate.com/bot)1 Why your site was visited
Domains are discovered from public sources only: Certificate Transparency logs, newly-registered-domain feeds, and public domain lists. We never use customer traffic or personal data to discover sites to crawl.
2 What it does on your site
- Fetches your home page over HTTPS (falling back to HTTP) and reads the
<title>, meta description, visible text and public metadata (JSON-LD,og:tags) to determine a category. That's it. It does not deep-crawl your site. - Stores extracted signals, not copies of your pages.
- Does not submit forms, log in, make purchases, execute payment flows, or harvest e-mail addresses or personal data.
- Polite by design: one identified User-Agent, a per-site rate limit (about one request every couple of seconds, coordinated across our workers), a hard per-request timeout and size cap, and infrequent revisits (only to refresh a category).
3 Controlling it with robots.txt
EnforceGateBot reads and respects /robots.txt (RFC 9309), including
Disallow rules for our bot and Crawl-delay. To block it entirely:
User-agent: EnforceGateBot
Disallow: /To only slow it down, set a Crawl-delay. Changes take effect on our next visit.
One honest note: blocking EnforceGateBot via robots.txt removes your
site from content-based categorization only. It does not hide a site from our
safety classification (malware / phishing), which is driven by independent
threat-intelligence sources rather than by crawling your page, so a malicious operator cannot evade
a block simply by disallowing the bot.
4 How to verify it's really EnforceGateBot
We do not publish a fixed IP list: egress addresses rotate as crawl capacity
scales. Verify a visitor claiming to be EnforceGateBot by the User-Agent above and a
reverse-DNS check (FCrDNS): our crawler IPs reverse-resolve to hostnames under our infrastructure
(e.g. *.exosys.ai) and forward-confirm. A request presenting our User-Agent that fails
that check is not ours.
5 Contact, wrong classification & opt-out
Questions about the crawler, a classification you believe is wrong, or a request that we stop visiting your site: email intelbot@enforcegate.com; please include your domain and the relevant log lines (User-Agent + timestamp).
Misclassification reports are read and answered by a human; they are valuable to us and feed directly into review of the category data.
EnforceGateBot's classifications feed the curated category lists used by EnforceGate, a self-hosted secure web gateway for URL filtering and network access control, available in the Pro, Business and Enterprise editions.