A recent analysis of ~140 million websites uncovered the top three most-blocked SEO bots:
MJ12bot (Majestic) – Blocked by 6.49% of sites
SemrushBot – Blocked by 6.34%
AhrefsBot – Blo.cked by 6.31%
Blocking these bots primarily disrupts link indexing—tools can’t track outbound links from blocked pages. However, metrics like traffic estimates and keyword rankings remain unaffected, as they rely on alternative data sources.
Why Blocking Matters: Platform-Specific Impacts
Ahrefs: Blocking AhrefsBot disables the Page History feature (which tracks content changes) and removes sites from Yep.com, Ahrefs’ search engine.
Majestic & Semrush: Blocked crawlers degrade link-graph accuracy but leave core SEO metrics intact.
Methodology: The study analyzed robots.txt directives, including:
Explicit bot disallowances
Blanket blocks (e.g., Disallow: /)
Partial allowances after initial blocks
(Note: Firewall/IP-based blocks were excluded.)
Why MJ12bot Tops the Block List
Majestic’s crawler leads due to:
Distributed crawling: IPs are untraceable, raising trust issues.
Longevity: Older crawlers face accumulated blocks.
Niche user base: Fewer advocates to petition for access.
Deeper Insights: Subdomains & Targeted Blocks
When analyzing 461 million subdomain-level robots.txt files:
SemrushBot emerges as the most blocked (5.76%), followed by:
Dotbot (Moz): 5.34%
MJ12bot: 4.96%
BLEXBot: 4.88%
AhrefsBot: 4.67%
Targeted Blocks: Majestic’s bot is the most deliberately blocked (via explicit directives), indicating proactive exclusion by webmasters.
Trends by Site Authority and Category
High-DR Sites (DR >45): SemrushBot faces the most resistance.
Industry Variations:
Autos & Vehicles: 39% block SEO bots
Books & Literature: 27%
Real Estate: 17%
Related topics:
- AI Is Reshaping the Search Landscape
- What Are Backlinks on a Website and Why Do They Matter?
- 15 Methods to Optimize Your Company’s Website and Boost SEO
