大网数据广告活动
网站运营与故障问答

Facebook crawler with no user agent spamming our site in possible DoS attack

· 52CE 我爱测

Crawlers registered to Facebook (ipv6 ending in :face:b00c::1) were slamming our site, seeing 10s of thousands of hits in just 20 minutes. We noticed they didn't have a user agent in the header and implemented a rule on cloudflar…

问题原文

原帖链接 · L Martin · CC BY-SA 3.0

Crawlers registered to Facebook (ipv6 ending in :face:b00c::1) were slamming our site, seeing 10s of thousands of hits in just 20 minutes. We noticed they didn't have a user agent in the header and implemented a rule on cloudflare to protect ourselves.

It appears they've patched the crawler and added a user agent 'Externalhit/1.1' which is a recognised crawler. Now they're circumventing the rule, I'm seeing 11,000 hits in 15 minutes. Often multiple times to the same page! This is crippling our database. It's prevent customers from legitimately using the site.

We've implemented a broad block on all of Facebook's IPs in order to try and remedy this but we've likely already lost business because of it.

My question is: Has anyone seen this before? Any idea what's causing it? Is there a channel for getting a response from Facebook or is there a legal route we should go?

Link to our tweet: https://twitter.com/TicketSource/status/969148062290599937 Tried FB developers group, and Facebook rep and were directed to Support. Filed a ticket, no response.

Log sample:

2018-03-01 09:00:33 10.0.1.175 GET /dylanthomas - 443 - facebookexternalhit/1.1 - 200 0 0 5394 2a03:2880:30:7fcf:face:b00c:0:8000
2018-03-01 09:00:33 10.0.1.175 GET /dylanthomas - 443 - facebookexternalhit/1.1 - 200 0 0 5362 2a03:2880:30:afd1:face:b00c:0:8000
2018-03-01 09:00:33 10.0.1.175 GET /dylanthomas - 443 - facebookexternalhit/1.1 - 200 0 0 5378 2a03:2880:30:7fcf:face:b00c:0:8000
2018-03-01 09:00:33 10.0.1.175 GET /dylanthomas - 443 - facebookexternalhit/1.1 - 200 0 0 5425 2a03:2880:30:2fea:face:b00c:0:8000
2018-03-01 09:00:33 10.0.1.175 GET /dylanthomas - 443 - facebookexternalhit/1.1 - 200 0 0 5394 2a03:2880:30:2fea:face:b00c:0:8000
2018-03-01 09:00:33 10.0.1.175 GET /dylanthomas - 443 - facebookexternalhit/1.1 - 200 0 0 5659 2a03:2880:30:2fd8:face:b00c:0:8000
2018-03-01 09:00:33 10.0.1.175 GET /dylanthomas - 443 - facebookexternalhit/1.1 - 200 0 0 5659 2a03:2880:11:dff3:face:b00c:0:8000
2018-03-01 09:00:36 10.0.1.175 GET /whitedreamspremiere - 443 - facebookexternalhit/1.1 - 200 0 0 5048 2a03:2880:2020:bffb:face:b00c:0:8000
2018-03-01 09:00:36 10.0.1.175 GET /helioscollective - 443 - facebookexternalhit/1.1 - 200 0 0 4633 2a03:2880:3020:1ffd:face:b00c:0:8000
2018-03-01 09:00:36 10.0.1.175 GET /helioscollective - 443 - facebookexternalhit/1.1 - 200 0 0 4727 2a03:2880:3011:afc5:face:b00c:0:8000
2018-03-01 09:00:36 10.0.1.175 GET /helioscollective - 443 - facebookexternalhit/1.1 - 200 0 0 4977 2a03:2880:3020:1ffd:face:b00c:0:8000
2018-03-01 09:00:36 10.0.1.175 GET /event/FDMEJD - 443 - facebookexternalhit/1.1 - 200 0 0 4868 2a03:2880:2111:1ff9:face:b00c:0:8000

Edit2: These IPs are crawling as we've found URLs from our payment process. So they followed a link and ended up in a session only URL.

Edit3: Facebook appear to have confirmed the bug and are looking for a fix.

已采纳解答

原帖链接 · L Martin · CC BY-SA 3.0

https://developers.facebook.com/bugs/1894024420610804

Per the answer from Facebook, any page shared on Facebook should expect that if their content is shared Facebook crawlers will increase traffic 10-20x that number of shares.

This sounds like Facebook is scraping the content every single time it's accessed, with little to no caching in place.

In our case, while Facebook is probably good for advertising overall, this is an immense strain when you run a database intensive page that's shared. We'll have to rate limit the traffic on our end to prevent a denial of service attack. A resource intensive answer to Facebook's over active bot.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2018-03-01)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。