The DiscoverBot

Identification

This crawler identifies itself as:

WebAtlasDiscoverBot/1.9 (+https://mevasearch.com/discover-bot)

Purpose & Logic

Unlike our main indexer, DiscoverBot is a lightweight crawler focused on link discovery. It follows https:// links from already known pages to find new domains.

• Respects Robots.txt (WebAtlasDiscoverBot rules).
• Uses 3-Layer Filtering to avoid trackers, ads, and junk URLs.
• Optimized for low server load (max 512KB per stream).
• Strictly adheres to Crawl-delay directives.

Filtering Layers

To ensure a high-quality search engine, the bot automatically skips:

Layer 1: Blocklisted ad/social domains.
Layer 2: Technical paths (e.g., /admin, /wp-login, /api).
Layer 3: Heuristic ad parameters (e.g., gclid, utm_source, clickid).

Webmaster Control

DiscoverBot shares the same control signals as our main bot. To block it, use:

User-agent: WebAtlasDiscoverBot
Disallow: /