大网数据广告活动
网站运营与故障问答

Why is this mobile/image Googlebot crawl rate so fast? (10-30 req per sec)

· 52CE 我爱测

This morning in logs I noticed a user got rate limited by our throttler 3 times within 1 second (429 too many requests error) at 3:30:11 AM EST: Mozilla/5.0 (Linux; Android 6.0; GCE x86 phone Build/MRA59G.MZC35) AppleWebKit/537.3…

问题原文

原帖链接 · dhaupin · CC BY-SA 3.0

This morning in logs I noticed a user got rate limited by our throttler 3 times within 1 second (429 too many requests error) at 3:30:11 AM EST: Mozilla/5.0 (Linux; Android 6.0; GCE x86 phone Build/MRA59G.MZC35) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2679.0 Safari/537.36 at IP 104.196.83.207. I've only seen Google on this IP once before, so I'm unsure what it's purpose is.

Upon looking it up it seems it's a Googlebot indeed. Noted below by Analog, its coming from Google Cloud. The last paths it was trying before being limited for 30 seconds were images. We have never had problems with Google speeding the site until last night. Our throttler is set to trigger a 30 sec 429 error if a user hits 10 requests in a 5 second window (session var based). 3 logs imply they hit this limit 3 times within 1 second (30 total requests per sec).

Right now, the Search Console setting says it's on the "let Google decide" -- clicking the radio to change rate puts it square in the middle at "1 request every 10 seconds". On its maximum setting it allows 2 requests a second (ie throttle zone).

I will adjust the crawl rate manually to 1 every 10 seconds, but in the meantime why is this bot crawling so fast, and why is it on android mobile crawling only images?

已采纳解答

原帖链接 · Analog · CC BY-SA 3.0

It is my understand that all google bot crawler IP addresses can be identified by the RDNS which is set to something along the lines of "crawl-(IP ADDRESS).googlebot.com".

The useragent you mentioned is also not used by the googlebot. Google lists the useragent strings of all their bots here.

The IP address you posted is part of the netblock used by Google Cloud customers as seen in the ASN comments

NetRange:       104.196.0.0 - 104.199.255.255
CIDR:           104.196.0.0/14
NetName:        GOOGLE-CLOUD
NetHandle:      NET-104-196-0-0-1
Parent:         NET104 (NET-104-0-0-0-0)
NetType:        Direct Allocation
OriginAS:       AS15169
Organization:   Google Inc. (GOOGL-2)
RegDate:        2014-08-27
Updated:        2015-09-21
Comment:        ** The IP addresses under this netblock are in use by Google Cloud customers ** 
Comment:        
Comment:        Direct all copyright and legal complaints to 
Comment:        https://support.google.com/legal/go/report
Comment:        
Comment:        Direct all spam and abuse complaints to 
Comment:        https://support.google.com/code/go/gce_abuse_report
Comment:        
Comment:        For fastest response, use the relevant forms above.
Comment:        
Comment:        Complaints can also be sent to the GC Abuse desk 
Comment:        ([email omitted]) 
Comment:        but may have longer turnaround times.
Comment:        
Comment:        Complaints sent to any other POC will be ignored.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2016-03-15)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。