问题原文
原帖链接 · tattvamasi · CC BY-SA 3.0
The problem is a whole load of 404 errors, as reported by Google Webmaster Tools, with pages and queries that have never been there. One of them is viewtopic.php, and I've also noticed a scary number of attempts to check if the site is a WordPress site (wp_admin) and for the cPanel login. I block TRACE already, and the server is equipped with some defense against scanning/hacking. However, this doesn't seem to stop. The referrer is, according to Google Webmaster, totally.me.
I have looked for a solution to stop this, because it isn't certainly good for the poor real actual users, let alone the SEO concerns.
I am using the Perishable Press mini black list (found here), a standard referrer blocker (for porn, herbal, casino sites), and even some software to protect the site (XSS blocking, SQL injection, etc). The server is using other measures as well, so one would assume that the site is safe (hopefully), but it isn't ending.
Does anybody else have the same problem, or am I the only one seeing this? Is it what I think, i.e., some sort of attack? Is there a way to fix it, or better, prevent this useless resource waste?
EDIT I've never used the question to thank for the answers, and hope this can be done. Thank you all for your insightful replies, which helped me to find my way out of this. I have followed everyone's suggestions and implemented the following:
- a honeypot
- a script that listens to suspect urls in the 404 page and sends me an email with user agent/ip, while returning a standard 404 header
- a script that rewards legitimate users, in the same 404 custom page, in case they end up clicking on one of those urls. In less than 24 hours I have been able to isolate some suspect IPs, all listed in Spamhaus. All the IPs logged so far belong to spam VPS hosting companies.
Thank you all again, I would have accepted all answers if I could.
已采纳解答
原帖链接 · Stephen Ostermiller · CC BY-SA 3.0
I often see another site that links to tons of pages on my site that don't exist. Even if you are clicking on that page and not seeing the link:
- The site might previously have had those links
- The site may be cloaking and serving those links only to Googlebot and not to visitors
It is a waste of resources, but it won't confuse Google and it won't hurt your rankings. Here is what Google's John Mueller (who works on Webmaster Tools and Sitemaps) has to say about 404 errors that appear in Webmaster tools:
HELP! MY SITE HAS 939 CRAWL ERRORS!!1
I see this kind of question several times a week; you’re not alone - many websites have crawl errors.
- 404 errors on invalid URLs do not harm your site’s indexing or ranking in any way. It doesn’t matter if there are 100 or 10 million, they won’t harm your site’s ranking. http://googlewebmastercentral.blogspot.ch/2011/05/do-404s-hurt-my-site.html
- In some cases, crawl errors may come from a legitimate structural issue within your website or CMS. How you tell? Double-check the origin of the crawl error. If there's a broken link on your site, in your page's static HTML, then that's always worth fixing. (thanks +Martino Mosna)
- What about the funky URLs that are “clearly broken?” When our algorithms like your site, they may try to find more great content on it, for example by trying to discover new URLs in JavaScript. If we try those “URLs” and find a 404, that’s great and expected. We just don’t want to miss anything important (insert overly-attached Googlebot meme here). http://support.google.com/webmasters/bin/answer.py?answer=1154698
- You don’t need to fix crawl errors in Webmaster Tools. The “mark as fixed” feature is only to help you, if you want to keep track of your progress there; it does not change anything in our web-search pipeline, so feel free to ignore it if you don’t need it. http://support.google.com/webmasters/bin/answer.py?answer=2467403
- We list crawl errors in Webmaster Tools by priority, which is based on several factors. If the first page of crawl errors is clearly irrelevant, you probably won’t find important crawl errors on further pages. http://googlewebmastercentral.blogspot.ch/2012/03/crawl-errors-next-generation.html
- There’s no need to “fix” crawl errors on your website. Finding 404’s is normal and expected of a healthy, well-configured website. If you have an equivalent new URL, then redirecting to it is a good practice. Otherwise, you should not create fake content, you should not redirect to your homepage, you shouldn’t robots.txt disallow those URLs -- all of these things make it harder for us to recognize your site’s structure and process it properly. We call these “soft 404” errors. http://support.google.com/webmasters/bin/answer.py?answer=181708
- Obviously - if these crawl errors are showing up for URLs that you care about, perhaps URLs in your Sitemap file, then that’s something you should take action on immediately. If Googlebot can’t crawl your important URLs, then they may get dropped from our search results, and users might not be able to access them either.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
