大网数据广告活动
网站运营与故障问答

Googlebot flooding server with requests for junk URLs with random data

· 52CE 我爱测

I'm having some trouble with GoogleBot. It keeps requesting a random URL that doesn't exist. It is trying to access: www.example.com/index.php/{TOKEN} That {TOKEN} is really random, no idea where it came from. I'm trying to respo…

问题原文

原帖链接 · Arthur lauck do nascimento · CC BY-SA 3.0

I'm having some trouble with GoogleBot. It keeps requesting a random URL that doesn't exist. It is trying to access:www.example.com/index.php/{TOKEN}

That {TOKEN} is really random, no idea where it came from. I'm trying to respond that the pages don't exist by 301 redirecting to home page (not sure if this is a good idea).

This is causing my server to overload, because it is TONS OF REQUESTS! What should I do to stop this?

Access Log:

example.com 66.249.64.28 - - [21/Feb/2018:12:13:48 -0300] "GET /index.php/66t-2nkznwh_91f4690bjij1wbgziq- HTTP/1.1" 301 178 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"

已采纳解答

原帖链接 · George · CC BY-SA 3.0

  • "What should I do". As an immediate action I'd set a rule in web-server config (e.g. .htaccess) to respond with 404 to that. 404 is in case you have no /index.php as a valid path on your server. Dong so will at least drop load from your interpreter (I assume it's PHP).
  • Next I'd put a rule in robots.txt to forbid such a path from indexing. It should completely stop Google from crawling those URIs and stop spending crawl budget on them.
  • After that I'd search for links to your site using one of those URIs. Who knows, maybe it will help you to find the reason where those links are coming to Google from. What if that's your own site?

That's it I think.

PS 301 is not a good idea I think. From my experience Bot will be coming back from time to time to confirm that redirection is still there. I guess that's not what you want. Moreover 404 is really fits better per definition.

a random URL that doesn't exist

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2018-02-21)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。