FEED & SITEMAP CRAWLER

The RssSitemapBot

Following the feeds. WebAtlasRssSitemapBot reads RSS, Atom, and sitemap files to feed our global indexing queue.

Identification

This crawler identifies itself as:

WebAtlasRssSitemapBot/1.0 (+https://mevasearch.com/rsssitemap-bot)

→ See also: WebAtlasDiscoverBot (link discovery)

Purpose & Logic

RssSitemapBot is a lightweight, focused crawler with a single job: read previously-submitted RSS/Atom feeds and sitemap files, and extract the URLs listed inside them.

Filtering Layers

To ensure a high-quality search engine, the bot automatically skips:

Layer 1: Blocklisted ad/social/tracking domains.
                    Layer 2: Technical paths (e.g., /admin, /wp-login, /api).
                    Layer 3: Heuristic ad parameters (e.g., gclid, utm_source, clickid).