大网数据广告活动
网站运营与故障问答

Page blocked by robots.txt showing up in site: search results with a description that is a mix of Chinese, English, and German

· 52CE 我爱测

I found a strange search result for a resource blocked by robots.txt . Why ist there the Chinese (guessed) text followed by the text Hello nighthawk! . Is this an esteregg of Google? Yesterday I tried to remove the URL from googl…

问题原文

原帖链接 · Mark · CC BY-SA 3.0

I found a strange search result for a resource blocked by robots.txt. Why ist there the Chinese (guessed) text followed by the text Hello nighthawk!. Is this an esteregg of Google?

Yesterday I tried to remove the URL from google with Webmaster Tools. There was no Hello Nighthawk!, only the 'blocked by robots.txt' message. The issue was reported by a co-woker.

This is the content of the robots.txt:

User-agent: * Disallow: /en

The domains gets redirected in the following way:

www.domain.com/en -> (301) https://domain.com/en

The page https://domain.com/en show the normal page with the correct title of the page.

The title of domain.com/en does not contain any of the words. I have searched the whole project to find the word 'nighthawk'. It is not included. And we never had any Chinese translations.

已采纳解答

原帖链接 · Stephen Ostermiller · CC BY-SA 3.0

Google includes uncrawlable pages in the index when they are linked from other sites.

That means that a link to the website like<a href="domain.com/en">[CHINESE] - Hey nighthawk</a> can show up in the search results.

Some have suggested that such occurrences are temporary. They aren't always. Google indexes uncrawlable pages because sometimes important pages are blocked by robots.txt. Matt Cutts explains:

You might wonder why Google will sometimes return an uncrawled url reference, even if Googlebot was forbidden from crawling that url by a robots.txt file. There’s a pretty good reason for that: back when I started at Google in 2000, several useful websites (eBay, the New York Times, the California DMV) had robots.txt files that forbade any page fetches whatsoever. Now I ask you, what are we supposed to return as a search result when someone does the query [california dmv]? We’d look pretty sad if we didn’t return www.dmv.ca.gov as the first result. But remember: we weren’t allowed to fetch pages from www.dmv.ca.gov at that point. The solution was to show the uncrawled link when we had a high level of confidence that it was the correct link.

You are unlikely to see this page from your search result except for the site: query that you did. Otherwise somebody would have to search for [CHINESE] Hey nighthawk or some portion thereof.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2016-10-20)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。