Blocking Marketing Spiders from Crawling the Site

·(edited)· / , , ·

648

reads·

likes

This article was last modified on . Some content may be outdated. Feel free to ask the author if you have questions.

AI Translation中文English

Key Insights

AI · GEN

Today I checked the site’s access logs and found a spider with the User Agent Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html) crawling the site very frequently, and judging by the 20-plus MB of logs it’s been doing this for quite a while.

Judging from the pages it crawls, it seems to randomly combine parameters from previously crawled pages and then re-crawl them, so the URLs are extremely long and almost all return 404. Coupled with requests every few seconds, normal access records in the logs are basically drowned out by tens of thousands of lines of spam spider entries.

At first I thought that since it’s a spider it should respect robots.txt, so I tried adding rules to robots.txt:

User-agent: SemrushBot
Disallow: /

After a quick search online, however, I discovered that this thing apparently doesn’t obey robots.txt 😅 (the official page claims the spider strictly follows robots.txt, but user reports say otherwise), and it’s not just SemrushBot; many marketing spiders ignore robots.txt. The only option left was to block them with Nginx. In Baota’s Nginx Free Firewall, under Global Config, click Rules in User-Agent Filter, and add the following regular expression (compiled from the web; I didn’t expect so many, and I threw in some useless UAs as well—check before use):

(nmap|NMAP|HTTrack|sqlmap|Java|zgrab|Go-http-client|CensysInspect|leiki|webmeup|Python|python|curllCurl|wget|Wget|toutiao|Barkrowler|AhrefsBot|a Palo Alto|ltx71|censys|DotBot|MauiBot|MegaIndex.ru|BLEXBot|ZoominfoBot|ExtLinksBot|hubspot|FeedDemon|Indy Library|Alexa Toolbar|AskTbFXTV|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|Bytespider|Ezooms|JikeSpider|SemrushBot)

After a few seconds you can see the blocked requests.

Later, watching the interception counter climb, I realized that whether it returns 404 or 444, every request still consumes server resources, so a request every few seconds isn’t sustainable. Digging deeper, I saw that these spiders all come from overseas IPs, and my site’s foreign DNS already points to Cloudflare, so I could let Cloudflare block them before they ever reach the server.

In Cloudflare’s dashboard for the domain, go to Security → WAF, click Add Rule. In the Expression Preview, add the following expression (again, check for any UAs you actually need):

CodeBlock Loading...

Then choose Block as the action and save.

From here, if Baota’s Nginx Free Firewall risk-interception counter stops rising while Cloudflare’s firewall rule counter skyrockets, it proves the garbage spiders are now being stopped at Cloudflare.

Woke-up update

After a nap, over two thousand requests had already been blocked—this thing is relentless 😅.

(http.user_agent contains "SemrushBot") or (http.user_agent contains "FeedDemon") or (http.user_agent contains "Indy Library") or (http.user_agent contains "Alexa Toolbar") or (http.user_agent contains "AskTbFXTV") or (http.user_agent contains "AhrefsBot") or (http.user_agent contains "CrawlDaddy") or (http.user_agent contains "CoolpadWebkit") or (http.user_agent contains "Java") or (http.user_agent contains "Feedly") or (http.user_agent contains "UniversalFeedParser") or (http.user_agent contains "ApacheBench") or (http.user_agent contains "Microsoft URL Control") or (http.user_agent contains "Swiftbot") or (http.user_agent contains "ZmEu") or (http.user_agent contains "jaunty") or (http.user_agent contains "Python-urllib") or (http.user_agent contains "lightDeckReports Bot") or (http.user_agent contains "YYSpider") or (http.user_agent contains "DigExt") or (http.user_agent contains "HttpClient") or (http.user_agent contains "MJ12bot") or (http.user_agent contains "heritrix") or (http.user_agent contains "Bytespider") or (http.user_agent contains "Ezooms") or (http.user_agent contains "JikeSpider") or (http.user_agent contains "HTTrack") or (http.user_agent contains "Apache-HttpClient") or (http.user_agent contains "harvest") or (http.user_agent contains "audit") or (http.user_agent contains "dirbuster") or (http.user_agent contains "pangolin") or (http.user_agent contains "nmap") or (http.user_agent contains "sqln") or (http.user_agent contains "hydra") or (http.user_agent contains "libwww") or (http.user_agent contains "BBBike") or (http.user_agent contains "sqlmap") or (http.user_agent contains "w3af") or (http.user_agent contains "owasp") or (http.user_agent contains "Nikto") or (http.user_agent contains "fimap") or (http.user_agent contains "havij") or (http.user_agent contains "BabyKrokodil") or (http.user_agent contains "netsparker") or (http.user_agent contains "httperf") or (http.user_agent contains " SF/") or (http.user_agent contains "zgrab") or (http.user_agent contains "NMAP") or (http.user_agent contains "Go-http-client") or (http.user_agent contains "CensysInspect") or (http.user_agent contains "leiki") or (http.user_agent contains "webmeup") or (http.user_agent contains "Python") or (http.user_agent contains "python") or (http.user_agent contains "wget") or (http.user_agent contains "Wget") or (http.user_agent contains "toutiao") or (http.user_agent contains "Barkrowler") or (http.user_agent contains "a Palo Alto") or (http.user_agent contains "ltx71") or (http.user_agent contains "censys") or (http.user_agent contains "DotBot") or (http.user_agent contains "MauiBot") or (http.user_agent contains "MegaIndex.ru") or (http.user_agent contains "BLEXBot") or (http.user_agent contains "ZoominfoBot") or (http.user_agent contains "ExtLinksBot") or (http.user_agent contains "hubspot")