大网数据广告活动
网站运营与故障问答

Why are bots repeatedly crawling pages of my site that I never created?

· 52CE 我爱测

In my logs, I noticed that a lot of requests were coming from different bots and web-crawlers, but what confused me was that they were visiting pages such as /fpss/track/73/ that definitely do not exist on the site. And there wer…

问题原文

原帖链接 · Run_Script · CC BY-SA 4.0

In my logs, I noticed that a lot of requests were coming from different bots and web-crawlers, but what confused me was that they were visiting pages such as /fpss/track/73/ that definitely do not exist on the site.

And there were hundreds if not thousands of variations of these non-existent URLs, meaning that I was getting about 5000 requests a day and only about 250 of them were actually real visitors on real URLs. I was also confused when I noticed that when visiting these URLs, instead of returning a 404 error as I would have expected, the HTML of the homepage showed up, but without any styles or javascript.

A possible solution for me could be for example to block robots from visiting /fpss/ in the robots.txt file, but I would like to understand why they are doing this in the first place.

已采纳解答

原帖链接 · Stephen Ostermiller · CC BY-SA 4.0

Web crawlers crawl URLs because they find links to them. There are several possible common reasons, but it is hard to narrow it down further without knowing more about your site.

An old site

These URLs could be from a previous owner of your domain name, or from a previous version of the site that you have since taken down. Once bots discover URLs, they rarely forget them. Search engine bots such as Googlebot crawl URLs that haven't worked in decades.

Broken links or misconfiguration

Your site itself may be creating broken links. Sometimes a content management system (CMS) or a plugin for it will automatically create links in ways you don't expect.

Malware

Your site could be infected by malware that automatically creates the links. Even once malware is cleaned up, bots may continue to crawl the URLs it created.

Link spam

Some other site may be linking to your site in weird ways. It could be a mistake on their part, maybe they got the domain name wrong. It could be a malicious attempt to get your penalized by search engines.

Crawler hueristics

Googlebot is known to try to find links in JavaScript. It often interprets random bits as URLs even when they are not. For example having JavaScript such as s = '/fpss/track/73/' would cause Googlebot to crawl your example URL because the string literal has slashes in it.


As a next step you should make sure the URLs return the proper "404 Not Found" status rather than serve up the contents of your home page. You probably have some rewrite rule that is causing that behavior. You'll need to remove that rule. It is possible that malware created that rule. If so, you will need to remove that malware by installing a clean version of your site on your server.

Google Search Console or Bing Webmaster Tools may be able to tell you where their crawlers are finding the links to those URLs. Verify your site with those services and see if either of them reports a referrer for any of those URLs.

Blocking the URLs in robots.txt might be OK, but I wouldn't recommend it unless the crawling is causing your server to bog down. Search engines occasionally index URLs that are blocked by robots.txt. If they can crawl the URLs and see a proper error status instead, there will be no danger of having those URLs indexed. Once the URLs return an error status, they will put much less strain on your server anyway.

I would also suggest linking to your CSS and JS using root relative linking. When the hrefs start with a slash (/), then they will load properly even for files in a subdirectory.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2020-01-02)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。