大网数据广告活动
网站运营与故障问答

Why are Bing and SEMRushBot ignoring crawl-delay in my robots.txt file?

· 52CE 我爱测

We host a large number of sites that have a large number of catalog pages. We would like to slow down several bots because the traffic is quite excessive from those bots. Specifically, we get quite a bit of traffic from BingBot a…

问题原文

原帖链接 · ChipnCharge · CC BY-SA 4.0

We host a large number of sites that have a large number of catalog pages. We would like to slow down several bots because the traffic is quite excessive from those bots. Specifically, we get quite a bit of traffic from BingBot and SEMRushBot that we'd like to slow down. The information pages for both bots say that they obey the crawl-delay directive. However, despite changing the crawl delays for both, I am seeing no change in traffic even after several days. Is there something wrong with my file? (I put 60 into the SEMRushBot delay but I read that they only delay for a max of 10 seconds. I have seen no change at all after adding them to the robots.txt file).

User-agent: *
Disallow: /nobots/
Disallow: /products/features/
Disallow: /product/features/
Disallow: /product/reviews/
Disallow: /webservices/ajax/
User-agent: yahoo-mmcrawler
Disallow: /m/
User-agent: MJ12bot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot
Crawl-delay: 60
User-agent: Bingbot
Crawl-delay: 10
Disallow: /nobots/
Disallow: /products/features/
Disallow: /product/features/
Disallow: /product/reviews/
Disallow: /webservices/ajax/
User-agent: dotbot
Crawl-delay: 1
User-agent: Goodzer
Crawl-delay: 1
User-agent: rogerbot
Crawl-delay: 5
User-agent: Baiduspider
Disallow: /
User-agent: YandexBot
Disallow: /
User-agent: YandexImages
Disallow: /
User-agent: Linguee Bot
Disallow: /
User-agent: Seekport Crawler
Disallow: /
User-agent: GrapeshotCrawler
Crawl-Delay: 1
User-agent: istellabot
Disallow: /
User-agent: SeznamBot
Disallow: /
Sitemap:

社区高分解答

原帖链接 · Stephen Ostermiller · CC BY-SA 4.0

Your robots.txt file does not do what (I think) you intend it to do because you do not use blank lines between the sections. The robots.txt standard says (emphasis added):

The format logically consists of a non-empty set or records, separated by blank lines. The records consist of a set of lines of the form:

<Field> ":" <value>

In this memo we refer to lines with a Field "foo" as "foo lines".

The record starts with one or more User-agent lines, specifying which robots the record applies to, followed by "Disallow" and "Allow" instructions to that robot. For example:

User-agent: webcrawler
User-agent: infoseek
Allow:    /tmp/ok.html
Disallow: /tmp
Disallow: /user/foo

So when you have multiple sections for different user agents, there need to be blank lines between them. You can also assign multiple user agents to the same rule. Your robots.txt should be:

User-agent: *
Disallow: /nobots/
Disallow: /products/features/
Disallow: /product/features/
Disallow: /product/reviews/
Disallow: /webservices/ajax/

User-agent: yahoo-mmcrawler
Disallow: /m/

User-agent: SemrushBot
Crawl-delay: 60

User-agent: Bingbot
Crawl-delay: 10
Disallow: /nobots/
Disallow: /products/features/
Disallow: /product/features/
Disallow: /product/reviews/
Disallow: /webservices/ajax/

User-agent: dotbot
User-agent: Goodzer
Crawl-delay: 1

User-agent: rogerbot
Crawl-delay: 5

User-agent: Baiduspider
User-agent: MJ12bot
User-agent: AhrefsBot
User-agent: YandexBot
User-agent: YandexImages
User-agent: Linguee Bot
User-agent: Seekport Crawler
User-agent: GrapeshotCrawler
User-agent: istellabot
User-agent: SeznamBot
Disallow: /

Keep in mind that if you provide a separate section for a bot, it no longer follows the User-agent: * directives. So Semrushbot, dotbot, Goodzer, and rogerbot are allowed to crawl your entire site including /nobots/. yahoo-mmcrawler is similarly allowed to crawl almost everything.

Even a crawl delay of 1 second significantly slows down the rate at which crawlers can fetch documents to the point that your server shouldn't notice additional load from them. A typical server can handle hundreds or even thousands of requests every second. I'd recommend putting a default Crawl-delay: 1 and not try to customize it for different bots. Most bots ignore it completely, but those that honor it should still be able to crawl hundreds of pages from your site in a reasonable time period. I think this robots.txt file would better match your intent and slow crawlers enough to prevent server overload:

User-agent: *
Crawl-delay: 1
Disallow: /nobots/
Disallow: /products/features/
Disallow: /product/features/
Disallow: /product/reviews/
Disallow: /webservices/ajax/

User-agent: yahoo-mmcrawler
Crawl-delay: 1
Disallow: /nobots/
Disallow: /products/features/
Disallow: /product/features/
Disallow: /product/reviews/
Disallow: /webservices/ajax/
Disallow: /m/

User-agent: Baiduspider
User-agent: MJ12bot
User-agent: AhrefsBot
User-agent: YandexBot
User-agent: YandexImages
User-agent: Linguee Bot
User-agent: Seekport Crawler
User-agent: GrapeshotCrawler
User-agent: istellabot
User-agent: SeznamBot
Disallow: /

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2020-07-02)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。