问题原文
原帖链接 · Milos Cuculovic · CC BY-SA 4.0
I would like to start using the Crawl-delay on the robots.txt and saw that the value is in seconds. Is there a way to set it up below 1 second. Maybe 0.1?
My goal is to allow crawlers not to run over 10 requests per second, so if they run one every 0.1sec this will fulfill my requirement.
Putting it to 1sec is too slow as in has to be done on a large website.
已采纳解答
原帖链接 · Stephen Ostermiller · CC BY-SA 4.0
As far as I know the Crawl-delay directive is only applicable to Bingbot. Googlebot does not use it. So this answer is going to focus on what Bing allows.
The crawl-delay directive accepts only positive, whole numbers as values.
So the smallest crawl delay you can set is 1. It does not support decimals such as 0.1.
If you want your site to be crawled at 10 requests per second, you should omit the Crawl-delay entirely and let bots decide on their own how fast they want to crawl your site.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
