问题原文
原帖链接 · ChipnCharge · CC BY-SA 4.0
We host a large number of sites that have a large number of catalog pages. We would like to slow down several bots because the traffic is quite excessive from those bots. Specifically, we get quite a bit of traffic from BingBot and SEMRushBot that we'd like to slow down. The information pages for both bots say that they obey the crawl-delay directive. However, despite changing the crawl delays for both, I am seeing no change in traffic even after several days. Is there something wrong with my file? (I put 60 into the SEMRushBot delay but I read that they only delay for a max of 10 seconds. I have seen no change at all after adding them to the robots.txt file).
User-agent: *
Disallow: /nobots/
Disallow: /products/features/
Disallow: /product/features/
Disallow: /product/reviews/
Disallow: /webservices/ajax/
User-agent: yahoo-mmcrawler
Disallow: /m/
User-agent: MJ12bot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot
Crawl-delay: 60
User-agent: Bingbot
Crawl-delay: 10
Disallow: /nobots/
Disallow: /products/features/
Disallow: /product/features/
Disallow: /product/reviews/
Disallow: /webservices/ajax/
User-agent: dotbot
Crawl-delay: 1
User-agent: Goodzer
Crawl-delay: 1
User-agent: rogerbot
Crawl-delay: 5
User-agent: Baiduspider
Disallow: /
User-agent: YandexBot
Disallow: /
User-agent: YandexImages
Disallow: /
User-agent: Linguee Bot
Disallow: /
User-agent: Seekport Crawler
Disallow: /
User-agent: GrapeshotCrawler
Crawl-Delay: 1
User-agent: istellabot
Disallow: /
User-agent: SeznamBot
Disallow: /
Sitemap:
社区高分解答
原帖链接 · Stephen Ostermiller · CC BY-SA 4.0
Your robots.txt file does not do what (I think) you intend it to do because you do not use blank lines between the sections. The robots.txt standard says (emphasis added):
The format logically consists of a non-empty set or records, separated by blank lines. The records consist of a set of lines of the form:
<Field> ":" <value>In this memo we refer to lines with a Field "foo" as "foo lines".
The record starts with one or more User-agent lines, specifying which robots the record applies to, followed by "Disallow" and "Allow" instructions to that robot. For example:
User-agent: webcrawler User-agent: infoseek Allow: /tmp/ok.html Disallow: /tmp Disallow: /user/foo
So when you have multiple sections for different user agents, there need to be blank lines between them. You can also assign multiple user agents to the same rule. Your robots.txt should be:
User-agent: *
Disallow: /nobots/
Disallow: /products/features/
Disallow: /product/features/
Disallow: /product/reviews/
Disallow: /webservices/ajax/
User-agent: yahoo-mmcrawler
Disallow: /m/
User-agent: SemrushBot
Crawl-delay: 60
User-agent: Bingbot
Crawl-delay: 10
Disallow: /nobots/
Disallow: /products/features/
Disallow: /product/features/
Disallow: /product/reviews/
Disallow: /webservices/ajax/
User-agent: dotbot
User-agent: Goodzer
Crawl-delay: 1
User-agent: rogerbot
Crawl-delay: 5
User-agent: Baiduspider
User-agent: MJ12bot
User-agent: AhrefsBot
User-agent: YandexBot
User-agent: YandexImages
User-agent: Linguee Bot
User-agent: Seekport Crawler
User-agent: GrapeshotCrawler
User-agent: istellabot
User-agent: SeznamBot
Disallow: /
Keep in mind that if you provide a separate section for a bot, it no longer follows the User-agent: * directives. So Semrushbot, dotbot, Goodzer, and rogerbot are allowed to crawl your entire site including /nobots/. yahoo-mmcrawler is similarly allowed to crawl almost everything.
Even a crawl delay of 1 second significantly slows down the rate at which crawlers can fetch documents to the point that your server shouldn't notice additional load from them. A typical server can handle hundreds or even thousands of requests every second. I'd recommend putting a default Crawl-delay: 1 and not try to customize it for different bots. Most bots ignore it completely, but those that honor it should still be able to crawl hundreds of pages from your site in a reasonable time period. I think this robots.txt file would better match your intent and slow crawlers enough to prevent server overload:
User-agent: *
Crawl-delay: 1
Disallow: /nobots/
Disallow: /products/features/
Disallow: /product/features/
Disallow: /product/reviews/
Disallow: /webservices/ajax/
User-agent: yahoo-mmcrawler
Crawl-delay: 1
Disallow: /nobots/
Disallow: /products/features/
Disallow: /product/features/
Disallow: /product/reviews/
Disallow: /webservices/ajax/
Disallow: /m/
User-agent: Baiduspider
User-agent: MJ12bot
User-agent: AhrefsBot
User-agent: YandexBot
User-agent: YandexImages
User-agent: Linguee Bot
User-agent: Seekport Crawler
User-agent: GrapeshotCrawler
User-agent: istellabot
User-agent: SeznamBot
Disallow: /
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
