问题原文
原帖链接 · Wold · CC BY-SA 4.0
My website, which typically has very low volume, suddenly received a huge spike in traffic all from the same user agent. We were receiving about 3 requests per second for a sustained period of about an hour all from the user agent "Facebot Twitterbot." The requests all look like this, to one URI:
Dec 16 00:35:36 website app/web.1 10.1.52.64 - - [16/Dec/2021:02:35:36 -0600] "GET /signup/ HTTP/1.1" 200 2299 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0"
We had a similar traffic event to another page in the past that lasted a bit longer about two weeks ago. This isn't a huge problem, since the site can handle the traffic and it seems to end after some time, but it does slow the response time by a few milliseconds typically and it spams our logs and makes it difficult to debug actual problems that crop up.
I have looked into this user agent a bit (see: here and here) it seems to be an IOS imessage utility function, though it it still unclear what exactly causes it. Is this traffic normal, should I have any reason to be concerned?
已采纳解答
原帖链接 · GuidedHacking · CC BY-SA 4.0
This is the iMessages app's crawler (from the users phone itself, not a Apple server).
Apple has chosen to use this useragent in their iMessages app to ensure the unfurling of the URL and the rich preview works more often than not. When someone sends a URL, their phone attempts to retrieve a picture from the website to use in the rich preview. They are assuming that the Facebook and Twitter crawlers are whitelisted by most websites, so they're abusing that to ensure the crawler's requests are not blocked.
The most common requests from this useragent are:
- /favicon.ico
- /apple-touch-icon.png
- /apple-touch-icon-precomposed.png
It's bizarre because no one blocks standard iPhone/Safari useragents, so the addition of facebookexternalhit/1.1 Facebot Twitterbot/1.0 doesn't really make sense to me.
It may also be Facebook messenger or a Facebook bot, but I haven't seen any real evidence for that except for some random posts online.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
