大网数据广告活动
网站运营与故障问答

Is Applebot following Crawl-Delay?

· 52CE 我爱测

Applebot have been crawling my website like crazy for weeks now, and it does not seems to slow down. Does it follow Crawl-Delay from robots.txt, or not?

问题原文

原帖链接 · magnetik · CC BY-SA 4.0

Applebot have been crawling my website like crazy for weeks now, and it does not seems to slow down.

Does it follow Crawl-Delay from robots.txt, or not?

社区高分解答

原帖链接 · Brac · CC BY-SA 4.0

To my knowledge and experience, Bing is the only bot to really observe the crawl-delay directive, but I saw that Yahoo and Yandex are supposed to also. This directive is pretty new and no standard have been defined, so its implementation varies from a bot to another as this guy says it here.

Google stated clearly it does not follow it and invites you to set this in your Search Console.

Since Apple says on its website that it respects standard robots.txt directives, I don't expect them to have implemented the crawl-delay directive in any way.

If the issue is still happening, I would block Applebot for a few days if you are sure it's the one causing too much load on your server.

One thing though, how did you identify it was Applebot the problem? Did you set up a log of all User Agents crawling your site? I did that once, and I was amazed with the high number of bots on my site in a single week.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2019-07-30)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。