ACTIVE CRAWLER

Meet WebAtlasBot

The official search engine crawler for MevaSearch, building a safe and filter-free map of the internet.

Identification

Our bot uses the following User-Agent string:

WebAtlasBot/2.0 (+https://mevasearch.com/bot)

Always verify the bot via reverse DNS to ensure the request originates from MevaSearch infrastructure.

Behavior

WebAtlasBot is designed to maintain a minimal load on web servers:

  • Respects robots.txt directives.
  • Supports Crawl-delay.
  • Verifies noindex and canonical tags.

WebAtlas Index

All crawled content is processed and stored in the WebAtlas Index, our proprietary global database designed for high-performance retrieval and semantic analysis.

By being part of the index, your site becomes discoverable through our search infrastructure.

Crawl Control

If you wish to restrict WebAtlasBot's access to your site, add the following rules to your robots.txt file:

User-agent: WebAtlasBot
Disallow: /private/
Crawl-delay: 5

The bot also respects HTML meta tags:

<meta name="robots" content="noindex">

Why are we visiting?

WebAtlasBot collects data to power the MevaSearch engine. We focus on high-quality, educational, and technical content. Our goal is not aggressive indexing, but rather creating a knowledge base resistant to algorithmic manipulation.