问题原文
原帖链接 · Arthur lauck do nascimento · CC BY-SA 3.0
I'm having some trouble with GoogleBot. It keeps requesting a random URL that doesn't exist. It is trying to access:www.example.com/index.php/{TOKEN}
That {TOKEN} is really random, no idea where it came from. I'm trying to respond that the pages don't exist by 301 redirecting to home page (not sure if this is a good idea).
This is causing my server to overload, because it is TONS OF REQUESTS! What should I do to stop this?
Access Log:
example.com 66.249.64.28 - - [21/Feb/2018:12:13:48 -0300] "GET /index.php/66t-2nkznwh_91f4690bjij1wbgziq- HTTP/1.1" 301 178 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"
已采纳解答
原帖链接 · George · CC BY-SA 3.0
- "What should I do". As an immediate action I'd set a rule in web-server config (e.g. .htaccess) to respond with
404to that. 404 is in case you have no/index.phpas a valid path on your server. Dong so will at least drop load from your interpreter (I assume it's PHP). - Next I'd put a rule in robots.txt to forbid such a path from indexing. It should completely stop Google from crawling those URIs and stop spending crawl budget on them.
- After that I'd search for links to your site using one of those URIs. Who knows, maybe it will help you to find the reason where those links are coming to Google from. What if that's your own site?
That's it I think.
PS 301 is not a good idea I think. From my experience Bot will be coming back from time to time to confirm that redirection is still there. I guess that's not what you want. Moreover 404 is really fits better per definition.
a random URL that doesn't exist
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
